Android Internals

July 5, 2014

Press ➡ or space to move to the next slide, 't' for ToC, 'h' for help.

2014 NewCircle, Inc. All rights reserved.

Overview

Android Stack

images/AndroidStack.svg

The purpose of this module is to explain the anatomy of the Android OS. We will walk through the Android stack, starting from the bottom and moving up:

Android Stack - The Details

images/Android_Stack.svg
Android Stack Details

Android Linux Kernel Layer

images/AndroidStack-Linux.svg
Android Linux Kernel Layer

Overview

The following are some of the changes/additions Android makes to the Linux kernel.

Binder IPC

Ashmem (Anonymous SHared MEMory)

Physical memory allocator

Buffer Sync Framework

Wakelocks

CPU Frequency Governor: Interactive

Alarm

Low Memory Killer (a.k.a. Viking Killer)

Logger

Paranoid Network Security

Other Kernel Changes

Android User-Space Native Layer

images/AndroidStack-NativeLayer.svg
Android User-Space Native Layer

Bionic Library

Changes From BSD libc

User-space Hardware Abstraction Layer (HAL)

Native Daemons

Flingers

Surface Flinger
Audio Flinger

Function Libraries

Application Runtime: Dalvik

images/Stack-Dalvik-Dan.jpg

Application Runtime: ART

$ oatdump --oat-file=system@priv-app@SystemUI.apk@classes.dex
MAGIC:
oat
007

CHECKSUM:
0x36096ec9

INSTRUCTION SET:
X86

...

OAT DEX FILE:
location: /system/priv-app/SystemUI.apk
checksum: 0x6e903ad8
0: Lcom/android/systemui/BatteryMeterView$1; (type_idx=324) (StatusInitialized)
1: Lcom/android/systemui/BatteryMeterView$BatteryTracker$1; (type_idx=325) (StatusInitialized)
  0: void com.android.systemui.BatteryMeterView$BatteryTracker$1.<init>(com.android.systemui.BatteryMeterView$BatteryTracker) (dex_method_idx=1200)
    DEX CODE:
      0x0000: iput-object v3, v2, Lcom/android/systemui/BatteryMeterView$BatteryTracker; com.android.systemui.BatteryMeterView$BatteryTracker$1.this$1 // field@142
      0x0002: invoke-direct {v2}, void java.lang.Object.<init>() // method@3809
      0x0005: const/4 v0, #+0
      0x0006: iput v0, v2, I com.android.systemui.BatteryMeterView$BatteryTracker$1.curLevel // field@137
      ...
      0x0020: return-void
    OAT DATA:
      frame_size_in_bytes: 48
      core_spill_mask: 0x000001e0 (r5, r6, r7, r8)
      fp_spill_mask: 0x00000000
      vmap_table: 0xf6f74131 (offset=0x0007f131)
      v2/r5, v0/r6, v1/r7, v65535/r8
      mapping_table: 0xf6f74118 (offset=0x0007f118)
      gc_map: 0xf6f74138 (offset=0x0007f138)
    CODE: 0xf6f74004 (offset=0x0007f004 size=276)...
      0xf6f74004:                 83EC2C        sub     esp, 44
      0xf6f74007:               896C2420        mov     [esp + 32], ebp
      ...

$ ll /data/dalvik-cache
-rw-r--r-- system   all_a4    1277824 2014-03-22 00:45 system@priv-app@Dialer.apk@classes.dex
...
-rw-r--r-- system   all_a8     528048 2014-03-22 00:45 system@priv-app@Launcher2.apk@classes.dex
...
-rw-r--r-- system   all_a7     469208 2014-03-22 00:45 system@priv-app@SystemUI.apk@classes.dex

$ file system@priv-app@SystemUI.apk@classes.dex
system@priv-app@SystemUI.apk@classes.dex: Dalvik dex file (optimized for host) version 036

$ ll /data/dalvik-cache
-rw-r--r-- system   all_a4    3531184 2014-03-24 01:25 system@priv-app@Dialer.apk@classes.dex
...
-rw-r--r-- system   all_a8    1651120 2014-03-24 01:24 system@priv-app@Launcher2.apk@classes.dex
...
-rw-r--r-- system   all_a7    1356208 2014-03-24 01:24 system@priv-app@SystemUI.apk@classes.dex

$ file system@priv-app@SystemUI.apk@classes.dex
system@priv-app@SystemUI.apk@classes.dex: ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, stripped

$ objdump -t system@priv-app@SystemUI.apk@classes.dex
system@priv-app@SystemUI.apk@classes.dex:     file format elf32-i386

SYMBOL TABLE:
no symbols


$ objdump -T system@priv-app@SystemUI.apk@classes.dex
system@priv-app@SystemUI.apk@classes.dex:     file format elf32-i386

DYNAMIC SYMBOL TABLE:
00001000 g    DO .rodata        0007f000 oatdata
00080000 g    DO .text  000cae57 oatexec
0014ae53 g    DO .text  00000004 oatlastword

Runtime Selection

images/Stack-Select-Runtime.png








Android Application Framework Layer

images/AndroidStack-ApplicationFramework.svg
Android Application Framework Layer

Overview

images/AndroidFooService.svg

Activity Manager Service

Package Manager Service

Power Manager Service

Alarm Manager Service

Notification Manager Service

Location Manager Service

Sensor Manager Service

Vibrator Manager Service

Connectivity Manager Service

Wifi Manager Service

Telephony Manager Service

Input Method Manager Service

Display Manager Service

Download Manager Service

Mount Service

Audio Manager Service

Window Manager Service

Device Policy Manager Service

Additional Manager Services

Android Applications Layer

images/AndroidStack-Applications.svg
Android Applications Layer

Android Built-in Applications

Android Built-in Content Providers

Android Built-in Input Methods

Android Built-in Wallpapers

Android Stack Review Questions

  1. What are the four layers of the Android Stack?

  2. What is the purpose of the Linux Kernel? How does Android use it?

  3. What’s missing in Android for it to be considered a more traditional Linux distribution?

  4. How is the Linux kernel on Android different?

  5. Name at least five Android Linux kernel extensions.

  6. What is Binder?

    1. What does Binder do and why do we need it?

    2. How is Binder exposed to the user-space?

    3. How does Binder compare to other similar mechanisms?

  7. What is Ashmem?

    1. Why do we need Ashmem?

    2. How do applications use Ashmem?

    3. How does Ashmem compare to other similar mechanisms?

  8. How does the power management on Android compare to traditional Linux distributions?

    1. What are wake locks?

    2. Where in the stack are the wake locks used?

    3. How do applications get access to wake locks?

    4. What is the purpose of early suspend feature of the Linux kernel?

  9. What is the purpose of the alarm driver?

  10. What are the two opposing objectives on Android when it comes with memory management?

    1. What is Android’s first line of defense when it comes to memory management?

    2. What is the low-memory-killer and what does it do?

    3. What are the relative levels of priority of application processes on Android?

    4. How does the low-memory-killer know who to kill and when?

    5. How can applications avoid being low-memory-killed?

  11. What is the purpose of the logger on Android?

    1. Name at least three log destinations.

    2. Where is the log information stored?

  12. What is the purpose of the CONFIG_ANDROID_PARANOID_NETWORK kernel option on Android?

  13. Name at least three sub-layers of the native layer.

  14. What is Bionic, why do we need it, and how does it differ from its alternatives?

  15. What are the two main purposes of user-space HAL on Android?

    1. How is the user-space HAL on Android exposed to the layers above it?

    2. Name at least three classes of devices exposed by user-space HAL.

  16. Name at least five native daemons and explain what they do.

    1. What is the purpose of ueventd on Android?

  17. What are the two "flingers" on Android?

  18. Name at least three "function" libraries on Android.

  19. What is the name of the Android’s media framework?

  20. What is the purpose of Dalvik on Android?

    1. How does Dalvik differ from its alternatives?

    2. What is the purpose of zygote?

    3. What was added to Dalvik in Froyo (Android 2.2)?

  21. What’s inside the Android Application Framework layer?

  22. What is the purpose of system services on Android? What value do they add?

  23. What is the purpose of managers for system services?

  24. What is the purpose of servicemanager daemon?

  25. Name at least five system services on Android.

  26. What does ActivityManagerService do?

  27. What does PackageManagerService do?

  28. What does the PowerManagerService do?

  29. What does the AlarmManagerService do?

  30. What does the KeyguardManagerService do?

  31. What does the InputMethodManagerService do?

  32. What is Android CDD and why do we need it?

  33. What’s inside an APK?

  34. What’s the difference between system and non-system apps?

  35. What are the different classes of applications that ship with Android?

  36. Name at least three "special" applications on Android?

Android Native Development Kit (NDK)

Objectives of NDK Module

Android is put together of about equal part Java and C. So, no wonder that we need an easy way to bridge between these two totally different worlds. Java offers Java Native Interface (JNI) as a framework connecting the world of Java to the native code. Android goes a step further by packaging other useful tools and libraries into a Native Development Kit, or NDK. NDK makes developing C/C++ code that works with an Android app much simpler than if one was to do it by hand. Topics covered include:

NDK in Action

images/Client-Native.svg
Using NDK to connect Activity to native code

Dalvik Runs Native

images/JNIonDalvik.svg

What is in NDK?

Why NDK?

Java Native Interface (JNI)

JNI Overview

In this module, we’ll explore the following topics:

JNI: What it is and why you’d care

JNI is an interface that allows Java to interact with code written in another language

Motivation for JNI:

JNI code is not portable!

JNI can also be used to invoke Java code from within natively-written applications - such as those written in C/C++.

In fact, the java command-line utility is an example of one such application, that launches Java code in a Java Virtual Machine.

JNI Components

JNI By Example

  1. We start by creating a Java class with one or more native methods

    src/com/marakana/jniexamples/Hello.java:
    package com.marakana.jniexamples;

public class Hello {

    public static native void sayHi(String who, int times); // 1

    static {
        System.loadLibrary("hello"); // 2
    }

    public static void main(String[] args) {
        sayHi(args[0], Integer.parseInt(args[1])); // 3
    }
}
    1 The method sayHi(String, int) will be implemented in C/C+ in separate files, which will be compiled into a shared library.
    2 Load the shared library by its logical name. The actual name is system-dependent: libhello.so (on Linux/Unix), hello.dll (on Windows), and libhello.jnilib (Mac OSX).
    3 Here we call our native method as a regular Java method.

  2. Compile the Java code 

    $ mkdir -p bin
$ javac -d bin/ src/com/marakana/jniexamples/Hello.java

  3. Using the javah tool, we generate the C header file from the compiled com.marakana.jniexamples.Hello class: 

    $ mkdir -p jni
$ javah -jni -classpath bin -d jni com.marakana.jniexamples.Hello

  4. Observe the generated C header file:

    jni/com_marakana_jniexamples_Hello.h:
    /* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class com_marakana_jniexamples_Hello */

#ifndef _Included_com_marakana_jniexamples_Hello
#define _Included_com_marakana_jniexamples_Hello
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class:     com_marakana_jniexamples_Hello
 * Method:    sayHi
 * Signature: (Ljava/lang/String;I)V
 */
JNIEXPORT void JNICALL Java_com_marakana_jniexamples_Hello_sayHi
  (JNIEnv *, jclass, jstring, jint);

#ifdef __cplusplus
}
#endif
#endif
    Method names resolve to C functions based on a pre-defined naming strategy: the prefix Java_, followed by a mangled fully-qualified class name, followed by an underscore ("_") separator, followed by a mangled method name. For overloaded native methods, two underscores ("__") followed by the mangled argument signature.

  5. Provide the C implementation:

    jni/com_marakana_jniexamples_Hello.c:
    #include "com_marakana_jniexamples_Hello.h"

JNIEXPORT void JNICALL Java_com_marakana_jniexamples_Hello_sayHi
  (JNIEnv *env, jclass clazz, jstring who, jint times) {
  const char *name = (*env)->GetStringUTFChars(env, who, NULL);
  if (name != NULL) {
    jint i;
    for (i = 0; i < times; i++) {
      printf("Hello %s\n", name);
    }
    (*env)->ReleaseStringUTFChars(env, who, name);
  }
}
    Most of the time, we cannot just use Java data types directly in C. For example, we have to convert java.lang.String to char * before we can effectively use it in C.
    This code assumes: #define NULL ((void *) 0)

  6. Compile the shared library 

    $ mkdir -p libs
$ gcc -o libs/libhello.jnilib -lc -shared \
    -I/System/Library/Frameworks/JavaVM.framework/Headers \
    jni/com_marakana_jniexamples_Hello.c
$ file libs/libhello.jnilib
libs/libhello.jnilib: Mach-O 64-bit dynamically linked shared library x86_64
    On Unix/Linux, compile as:
    gcc -o libs/libhello.so -lc -shared -fPIC -I$JAVA_HOME/include jni/com_marakana_jniexamples_Hello.c

  7. Run our code 

    $ java -Djava.library.path=libs -classpath bin com.marakana.jniexamples.Hello Student 5
Hello Student
Hello Student
Hello Student
Hello Student
Hello Student
    Instead of specifying -Djava.library.path=libs, we could have preceded our java command with export LD_LIBRARY_PATH=libs.
    Common mistakes resulting in java.lang.UnsatisfiedLinkError usually come from incorrect naming of the shared library (O/S-dependent), the library not being in the search path, or wrong library being loaded by Java code.

Native Method Arguments

Primitive Type Mapping

Reference Type Mapping

Global and Local References

Using Strings

Arrays

A note about memory

The pointer resulting from GetTypeArrayElements(…) is valid until ReleaseTypeArrayElements(…) is called (unless mode == JNI_COMMIT).

If isCopy is not NULL, then *isCopy is set to JNI_TRUE if a copy is made. When *isCopy == JNI_FALSE, the returned array is a direct pointer to the elements of the Java array, which is then pinned in memory.

Regardless of isCopy, we have to call ReleaseTypeArrayElements(…, int mode) when we are done using the native array, either to un-pin the Java array in memory when *isCopy == JNI_FALSE, or, when *isCopy == JNI_TRUE, to:

  • copy the native array over the Java array and free it (when mode == 0)
  • copy the native array over the Java array but not free it (when mode == JNI_COMMIT)
    This option assumes that we will call ReleaseTypeArrayElements(…, JNI_ABORT) at some later point
  • leave the Java array intact and free the native array (when mode == JNI_ABORT)

JNI also supports a critical version of these functions:

void * GetPrimitiveArrayCritical(JNIEnv *env, jarray array, jboolean *isCopy);
void ReleasePrimitiveArrayCritical(JNIEnv *env, jarray array, void *carray, jint mode);

The function GetPrimitiveArrayCritical(…) is similar to GetTypeArrayElements(…), except that the VM is more likely to return the pointer to the primitive array (i.e. *isCopy == JNI_FALSE is more likely). However, this comes with some caveats. The native code between GetPrimitiveArrayCritical(…) and ReleasePrimitiveArrayCritical(…) must not call other JNI functions, or make any system calls that may cause the current thread to block and wait for another Java thread. This is because the VM may temporarily suspend the garbage collection - in case it does not support memory pinning.

Reflection

Exceptions

Registering Native Methods On Load

Using NDK with C

images/NDK_Using_C.svg
NDK Process Using C

Using NDK with C++

images/NDK_Using_CPP.svg
NDK Process Using C++

NDK and JNI by Example

Fibonacci - Java Native Function Prototypes

We start off by defining C function prototypes as native Java methods (wrapped in some class):

FibonacciNative/src/com/marakana/android/fibonaccinative/FibLib.java
package com.marakana.android.fibonaccinative;

import android.util.Log;

public class FibLib {
    private static final String TAG = "FibLib";

    private static long fib(long n) {
        return n <= 0 ? 0 : n == 1 ? 1 : fib(n - 1) + fib(n - 2);
    }

    // Recursive Java implementation of the Fibonacci algorithm
    // (included for comparison only)
    public static long fibJR(long n) {
        Log.d(TAG, "fibJR(" + n + ")");
        return fib(n);
    }

    // Function prototype for future native recursive implementation
    // of the Fibonacci algorithm
    public native static long fibNR(long n);


    // Iterative Java implementation of the Fibonacci algorithm
    // (included for comparison only)
    public static long fibJI(long n) {
        Log.d(TAG, "fibJI(" + n + ")");
        long previous = -1;
        long result = 1;
        for (long i = 0; i <= n; i++) {
            long sum = result + previous;
            previous = result;
            result = sum;
        }
        return result;
    }

    // Function prototype for future iterative recursive implementation
    // of the Fibonacci algorithm
    public native static long fibNI(long n);

    static {
        // as defined by LOCAL_MODULE in Android.mk
        System.loadLibrary("com_marakana_android_fibonaccinative_FibLib");
    }
}

Fibonacci - Function Prototypes in a C Header File

We then extract our C header file with our function prototypes:

  1. On the command line, change to your project’s root directory 

    $ cd /path/to/workspace/FibonacciNative

  2. Create jni sub-directory 

    $ mkdir jni

  3. Extract the C header file from com.marakana.android.fibonaccinative.FibLib class: 

    $ javah -jni -classpath bin/classes -d jni com.marakana.android.fibonaccinative.FibLib
    Prior to ADT r14, compiled class files were kept directly in the bin/ directory, so in our javah command we would’ve used -classpath bin instead.

  4. Check out the resulting file:

    FibonacciNative/jni/com_marakana_android_fibonaccinative_FibLib.h
    /* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class com_marakana_android_fibonaccinative_FibLib */

#ifndef _Included_com_marakana_android_fibonaccinative_FibLib
#define _Included_com_marakana_android_fibonaccinative_FibLib
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class:     com_marakana_android_fibonaccinative_FibLib
 * Method:    fibNR
 * Signature: (J)J
 */
JNIEXPORT jlong JNICALL Java_com_marakana_android_fibonaccinative_FibLib_fibNR
  (JNIEnv *, jclass, jlong);

/*
 * Class:     com_marakana_android_fibonaccinative_FibLib
 * Method:    fibNI
 * Signature: (J)J
 */
JNIEXPORT jlong JNICALL Java_com_marakana_android_fibonaccinative_FibLib_fibNI
  (JNIEnv *, jclass, jlong);

#ifdef __cplusplus
}
#endif
#endif
    The function prototype names are name-spaced to the classname they are found in.

Fibonacci - Provide C Implementation

We provide the C implementation of com_marakana_android_fibonacci_FibLib.h header file:

FibonacciNative/jni/com_marakana_android_fibonaccinative_FibLib.c
/* Include the header file that was created via "javah -jni" command */
#include "com_marakana_android_fibonaccinative_FibLib.h"
#include <android/log.h>

/* Recursive implementation of the fibonacci algorithm (in a helper function) */
static jlong fib(jlong n) {
    return n <= 0 ? 0 : n == 1 ? 1 : fib(n - 1) + fib(n - 2);
}

/* Actual implementation of JNI-defined `fibNR` (recursive) function */
JNIEXPORT jlong JNICALL Java_com_marakana_android_fibonaccinative_FibLib_fibNR
  (JNIEnv *env, jclass clazz, jlong n) {
	__android_log_print(ANDROID_LOG_DEBUG, "FibLib.c", "fibNR(%lld)", n);
	return fib(n);
}

/* Actual implementation of JNI-defined `fibNI` (iterative) function */
JNIEXPORT jlong JNICALL Java_com_marakana_android_fibonaccinative_FibLib_fibNI
  (JNIEnv *env, jclass clazz, jlong n) {
	jlong previous = -1;
	jlong result = 1;
	jlong i;
	__android_log_print(ANDROID_LOG_DEBUG, "FibLib.c", "fibNI(%lld)", n);
	for (i = 0; i <= n; i++) {
		jlong sum = result + previous;
		previous = result;
		result = sum;
	}
	return result;
}

Fibonacci - An Alternative Implementation (C++)

We could also use an alternative mechanism of linking native-code to managed code by pre-registering our functions. This leads to earlier detection of method-function mismatch issues, a slight performance improvement, and spares us the redundancy of the header file and the use of the javah command.

FibonacciNative/jni/com_marakana_android_fibonaccinative_FibLib.cpp
#include <jni.h>
#include <android/log.h>

static jlong fib(jlong n) {
	return n <= 0 ? 0 : n == 1 ? 1 : fib(n - 1) + fib(n - 2);
}

static jlong fibNR(JNIEnv *env, jclass clazz, jlong n) {
	__android_log_print(ANDROID_LOG_DEBUG, "FibLib.c", "fibNR(%lld)", n);
	return fib(n);
}

static jlong fibNI(JNIEnv *env, jclass clazz, jlong n) {
	jlong previous = -1;
	jlong result = 1;
	jlong i;
	__android_log_print(ANDROID_LOG_DEBUG, "FibLib.c", "fibNI(%lld)", n);
	for (i = 0; i <= n; i++) {
		jlong sum = result + previous;
		previous = result;
		result = sum;
	}
	return result;
}

static JNINativeMethod method_table[] = {
	{ "fibNR", "(J)J", (void *) fibNR },
	{ "fibNI", "(J)J", (void *) fibNI }
};

extern "C" jint JNI_OnLoad(JavaVM* vm, void* reserved) {
	JNIEnv* env;
	if (vm->GetEnv(reinterpret_cast<void**>(&env), JNI_VERSION_1_6) != JNI_OK) {
		return JNI_ERR;
	} else {
		jclass clazz = env->FindClass(
				"com/marakana/android/fibonaccinative/FibLib");
		if (clazz) {
			jint ret = env->RegisterNatives(clazz, method_table,
					sizeof(method_table) / sizeof(method_table[0]));
			if (ret == 0) {
				return JNI_VERSION_1_6;
			}
		}
		return JNI_ERR;
	}
}
Most of the Android’s JNI-based shared libraries are built using this, "alternative", approach where the functions are pre-registered.

Fibonacci - Makefile

We need a Android.mk makefile, which will be used by NDK to compile our JNI code into a shared library:

FibonacciNative/jni/Android.mk
# Defines the root to all other relative paths
# The macro function my-dir, provided by the build system,
# specifies the path of the current directory (i.e. the
# directory containing the Android.mk file itself)
LOCAL_PATH := $(call my-dir)

# Clear all LOCAL_XXX variables with the exception of
# LOCAL_PATH (this is needed because all variables are global)
include $(CLEAR_VARS)

# List all of our C/C++ files to be compiled (header file
# dependencies are automatically computed)
LOCAL_SRC_FILES := com_marakana_android_fibonaccinative_FibLib.c

# The name of our shared module (this name will be prepended
# by lib and postfixed by .so)
LOCAL_MODULE := com_marakana_android_fibonaccinative_FibLib

# We need to tell the linker about our use of the liblog.so
LOCAL_LDLIBS += -llog

# Collects all LOCAL_XXX variables since "include $(CLEAR_VARS)"
#  and determines what to build (in this case a shared library)
include $(BUILD_SHARED_LIBRARY)
It’s easiest to copy the Android.mk file from another (sample) project and adjust LOCAL_SRC_FILES and LOCAL_MODULE as necessary
See /path/to/ndk-installation-dir/docs/ANDROID-MK.html for the complete reference of Android make files (build system)

Fibonacci - Compile Our Shared Module

Finally, from the root of our project (i.e. FibonacciNative/), we run ndk-build to build our code into a shared library (FibonacciNative/libs/armeabi/libcom_marakana_android_fibonacci_FibLib.so):

$ ndk-build
Compile thumb  : com_marakana_android_fibonaccinative_FibLib <= com_marakana_android_fibonaccinative_FibLib.c
SharedLibrary  : libcom_marakana_android_fibonaccinative_FibLib.so
Install        : libcom_marakana_android_fibonaccinative_FibLib.so => libs/armeabi/libcom_marakana_android_fibonaccinative_FibLib.so
The command ndk-build comes from the NDK’s installation directory (e.g. /path/to/android-ndk-r5b), so it’s easiest if we add this directory to our PATH.
On Windows, older version of NDK required Cygwin (a Unix-like environment and command-line interface for Microsoft Windows) to provide "shell" (bash) and "make" (gmake) to ndk-build.
Controlling CPU Application Binary Interface (ABI)

By default, the NDK will generate machine code for the armeabi - i.e. ARMv5TE with support for Thumb-1.

In addition to ARMv5, NDK also comes with the toolchains necessary to build code for:

  • ARMv7-A (including hardware FPU/VFPv3-D16, Thumb-2, VFPv3-D32/ThumbEE, and SIMD/NEON)
  • x86 (as of r6)
  • MIPS (as of r8)

We can explicitly select ABIs using APP_ABI variable:

$ ndk-build APP_ABI=armeabi
$ ndk-build APP_ABI=armeabi-v7a
$ ndk-build APP_ABI=x86
$ ndk-build APP_ABI=mips

We can also combine them, thereby building a "fat binary":

$ ndk-build "APP_ABI=armeabi armeabi-v7a x86 mips"
$ ndk-build APP_ABI=all

Finally, we can also persist our choice of ABI, by saving APP_ABI in a Application.mk file:

jni/Application.mk:
APP_ABI := all

Each ABI-specific library, gets packaged as lib/<ABI>/lib<name>.so inside our APK.

Upon installation, the library that best matches the native ABI of the device it will execute on, will get copied to /data/data/<package>/lib/lib<name>.so.

To remove all generated binaries, run:

$ ndk-build clean
Clean: com_marakana_android_fibonaccinative_FibLib [armeabi]
Clean: stdc++ [armeabi]

Fibonacci - Client

We can now build the "client" of our library (in this case a simple activity) to use our FibLib library.

Fibonacci - String Resources
FibonacciNative/res/values/strings.xml
<?xml version="1.0" encoding="utf-8"?>
<resources>
	<string name="hello">Get Your Fibonacci Here!</string>
	<string name="app_name">Fibonacci Native</string>
	<string name="input_hint">Enter N</string>
	<string name="input_error">Numbers only!</string>
	<string name="button_text">Get Fib Result</string>
	<string name="progress_text">Calculating…</string>
	<string name="fib_error">Failed to get Fibonacci result</string>
	<string name="type_fib_jr">fibJR</string>
	<string name="type_fib_ji">fibJI</string>
	<string name="type_fib_nr">fibNR</string>
	<string name="type_fib_ni">fibNI</string>
</resources>
Fibonacci - User Interface (Layout)
screens/FibonacciNativeMainLayout.png
Fibonacci Native Main Layout
FibonacciNative/res/layout/main.xml
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
	android:orientation="vertical" android:layout_width="fill_parent"
	android:layout_height="fill_parent">
	<TextView android:text="@string/hello" android:layout_height="wrap_content"
		android:layout_width="fill_parent" android:textSize="25sp" android:gravity="center"/>
	<EditText android:layout_height="wrap_content"
		android:layout_width="match_parent" android:id="@+id/input"
		android:hint="@string/input_hint" android:inputType="number"
		android:gravity="right" />
	<RadioGroup android:orientation="horizontal"
		android:layout_width="match_parent" android:id="@+id/type"
		android:layout_height="wrap_content">
		<RadioButton android:layout_height="wrap_content"
			android:checked="true" android:id="@+id/type_fib_jr" android:text="@string/type_fib_jr"
			android:layout_width="match_parent" android:layout_weight="1" />
		<RadioButton android:layout_height="wrap_content"
			android:id="@+id/type_fib_ji" android:text="@string/type_fib_ji"
			android:layout_width="match_parent" android:layout_weight="1" />
		<RadioButton android:layout_height="wrap_content"
			android:id="@+id/type_fib_nr" android:text="@string/type_fib_nr"
			android:layout_width="match_parent" android:layout_weight="1" />
		<RadioButton android:layout_height="wrap_content"
			android:id="@+id/type_fib_ni" android:text="@string/type_fib_ni"
			android:layout_width="match_parent" android:layout_weight="1" />
	</RadioGroup>
	<Button android:text="@string/button_text" android:id="@+id/button"
		android:layout_width="match_parent" android:layout_height="wrap_content" />
	<TextView android:id="@+id/output" android:layout_width="match_parent"
		android:layout_height="match_parent" android:textSize="20sp" android:gravity="center|top"/>
</LinearLayout>
Fibonacci - FibonacciActivity
FibonacciNative/src/com/marakana/android/fibonaccinative/FibonacciActivity.java
package com.marakana.android.fibonaccinative;

import java.util.Locale;

import android.app.Activity;
import android.app.ProgressDialog;
import android.os.AsyncTask;
import android.os.Bundle;
import android.os.SystemClock;
import android.text.TextUtils;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.Button;
import android.widget.EditText;
import android.widget.RadioGroup;
import android.widget.TextView;

public class FibonacciActivity extends Activity implements OnClickListener {

	private EditText input;

	private RadioGroup type;

	private TextView output;

	@Override
	public void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.main);
		this.input = (EditText) super.findViewById(R.id.input);
		this.type = (RadioGroup) super.findViewById(R.id.type);
		this.output = (TextView) super.findViewById(R.id.output);
		Button button = (Button) super.findViewById(R.id.button);
		button.setOnClickListener(this);
	}

	public void onClick(View view) {
		String s = this.input.getText().toString();
		if (TextUtils.isEmpty(s)) {
			return;
		}
		final ProgressDialog dialog = ProgressDialog.show(this, "",
				"Calculating...", true);
		final long n = Long.parseLong(s);
		final Locale locale = super.getResources().getConfiguration().locale;
		new AsyncTask<Void, Void, String>() {

			@Override
			protected String doInBackground(Void... params) {
				long result = 0;
				long t = SystemClock.uptimeMillis();
				switch (FibonacciActivity.this.type.getCheckedRadioButtonId()) {
				case R.id.type_fib_jr:
					result = FibLib.fibJR(n);
					break;
				case R.id.type_fib_ji:
					result = FibLib.fibJI(n);
					break;
				case R.id.type_fib_nr:
					result = FibLib.fibNR(n);
					break;
				case R.id.type_fib_ni:
					result = FibLib.fibNI(n);
					break;
				}
				t = SystemClock.uptimeMillis() - t;
				return String.format(locale, "fib(%d)=%d in %d ms", n, result,
						t);
			}

			@Override
			protected void onPostExecute(String result) {
				dialog.dismiss();
				FibonacciActivity.this.output.setText(result);
			}
		}.execute();
	}
}
For the sake of brevity, this client code has been kept simple, but its use of AsyncTask is brittle. In case of screen rotations, the activity will get re-created, and the task will not properly re-connect to its view.

Fibonacci - Result

screens/FibonacciNativeResult.png
Fibonacci Native Result

NDK’s Stable APIs

The header files for NDK stable APIs are available at /path/to/ndk/platforms/<android-platform>/<arch-name>/usr/include.

Android-specific Log Support

ZLib Compression Library

The OpenGL ES 1.x Library

The OpenGL ES 2.0 Library

The jnigraphics Library

The OpenSL ES native audio Library

The Android native application APIs

With the exception of the libraries listed above, the native system libraries in the Android platform are not considered "stable" and may change in future platform versions. Unless our library is being built for a specific Android ROM, we should only make use of the stable libraries provided by the NDK.
All the header files are available under: /path/to/ndk-installation-dir/platforms/android-9/arch-arm/usr/include/
See /path/to/ndk-installation-dir/docs/STABLE-APIS.html for the complete reference of NDK’s stable APIs.

Lab: NDK

The objective of this lab is to test your understanding of JNI and NDK. We will do so by adding JNI code to an existing application.

  1. Start by importing LogNative application into Eclipse

    1. Menu Bar → File → Import… → Git → Projects from Git → Next >

    2. Under Select Repository Source select URI → Next >

    3. Under Source Git Repository → Location → URI: enter https://github.com/thenewcircle/LogNative.git → Next >

    4. Under Branch Selection, leave all branches selected (checked) → Next >

    5. Under Local Destination → Destination specify directory of your choice (e.g. ~/android/workspace/LogNative) → Next >

    6. Under Select a wizard to use for importing projects, leave Wizard for project import as Import existing projects → Next >

    7. Under Import Projects → Projects, leave LogNative as selected (checked) → Finish

      This project can also be downloaded as a ZIP file

  2. Examine and test your project in Eclipse

    1. Run the application on a device/emulator

    2. Enter some tag and message to log, click on the Log button and observe via adb logcat that your message get logged (assuming Java was selected)

  3. Implement com.marakana.android.lognative.LogLib.logN(int priority, String tag, String msg) in C

    1. Mark the method as native

    2. Remove its body

    3. Extract its function prototype into a C header file (hint: javah)

    4. Implement the function by taking advantage of <android/log.h> (i.e. /system/lib/liblog.so)

    5. Provide the makefile(s)

  4. Build (via ndk-build)

  5. Run your application

  6. Test by selecting Native in the UI and checking that the log tag/message shows up in adb logcat

  7. As a bonus:

    1. Throw java.lang.NullPointerException if tag or msg are null

    2. Throw java.lang.IllegalArgumentException if priority is not one of the allowed types or if tag or msg are empty

Don’t forget to convert tag and msg strings from the Java format (jstring) to native format (char *) before trying to use them in int __android_log_write(int prio, const char *tag, const char *text). Be sure to free the native strings before returning from the native method. Finally, don’t forget to tell the linker about your use of the log library.

The solution is provided:

Android NDK and JNI Review Questions

  1. Who uses JNI on Android and why?

  2. What is the NDK?

  3. What is JNI?

  4. What is the purpose of jni.h?

  5. What is the purpose of javah tool?

  6. What are the first two arguments to all native functions that model Java methods declared as native?

  7. What is jlong and why do we need it (and its cousins)?

  8. How are boolean-s modeled in native code?

  9. How are java.lang.Object-s modeled in native code?

  10. What’s the definition of NULL?

  11. What is the difference between local and global references?

  12. What is the difference between Java strings and C "strings"?

  13. What is the difference between GetStringChars and GetStringUTFChars?

  14. How do modified UTF-8 strings differ from normal UTF-8 strings?

  15. Why do we have to call ReleaseStringChars or ReleaseStringUTFChars when we are done with the C strings?

  16. Name at least five array operations?

  17. What is the purpose of mode attribute in ReleaseByteArrayElements(…, mode) (and its cousins)?

  18. What is the signature of void f(int[] a, String s, long n)?

  19. What happens when an exception is thrown by native code?

  20. What is the significance of extern "C" jint JNI_OnLoad(JavaVM* vm, void* reserved) function?

  21. What can we do with jint RegisterNatives(JNIEnv*, jclass, const JNINativeMethod*, jint) function?

  22. What does LOCAL_MODULE specify in Android.mk?

  23. How do we control the ABI when compiling NDK code?

  24. Name at least three stable NDK APIs.

Summary of NDK Module

In this module, you learned how Android uses JNI to bridge between the world of Java and the native code. You also learned how NDK makes the process of working with JNI simpler by providing tools and framework for developing native libraries as well as packaging them with the app.

Android Binder Inter Process Communication (IPC) with AIDL

What is Binder?

images/Binder.svg

IPC

images/ipc.jpg

Why Binder?

images/AndroidStackArchitecture-with-connections.svg

IPC with Intents and ContentProviders?

images/HighLevelIPC.svg

Messenger IPC

images/MessengerIPC.svg



Binder Terminology

images/BinderTerminology.svg
Binder (Framework)

The overall IPC architecture

Binder Driver

The kernel-level driver that fascinates the communication across process boundaries

Binder Protocol

Low-level protocol (ioctl-based) used to communicate with the Binder driver

IBinder Interface

A well-defined behavior (i.e. methods) that Binder Objects must implement

AIDL

Android Interface Definition Language used to describe business operations on an IBinder Interface

Binder (Object)

A generic implementation of the IBinder interface

Binder Token

An abstract 32-bit integer value that uniquely identifies a Binder object across all processes on the system

Binder Service

An actual implementation of the Binder (Object) that implements the business operations

Binder Client

An object wanting to make use of the behavior offered by a binder service

Binder Transaction

An act of invoking an operation (i.e. a method) on a remote Binder object, which may involve sending/receiving data, over the Binder Protocol

Parcel

"Container for a message (data and object references) that can be sent through an IBinder." A unit of transactional data - one for the outbound request, and another for the inbound reply

Marshalling

A procedure for converting higher level applications data structures (i.e. request/response parameters) into parcels for the purposes of embedding them into Binder transactions

Unmarshalling

A procedure for reconstructing higher-level application data-structures (i.e. request/response parameters) from parcels received through Binder transactions

Proxy

An implementation of the AIDL interface that un/marshals data and maps method calls to transactions submitted via a wrapped IBinder reference to the Binder object

Stub

A partial implementation of the AIDL interface that maps transactions to Binder Service method calls while un/marshalling data

Context Manager (a.k.a. servicemanager)

A special Binder Object with a known handle (registered as handle 0) that is used as a registry/lookup service for other Binder Objects (name → handle mapping)

Binder Communication and Discovery

Location Service: An Example

images/LocationManagerServiceArchitecture.svg

AIDL

Binder Object Reference Mapping Across Process Boundaries

images/BinderReferences.svg

Building a Binder-based Service and Client by Example

images/FibonacciClientServiceArchitecture.svg

FibonacciCommon - Define AIDL Interface and Custom Types

FibonacciService - Implement AIDL Interface and Expose It To Our Clients

Expose our AIDL-defined Service Implementation to Clients

FibonacciClient - Using AIDL-defined Binder-based Services

Async Binder IPC (by Example)

FibonacciCommon - Defining a oneway AIDL Service

FibonacciService - Implementing our async AIDL service

FibonacciClient - Implementing our async AIDL client

Sharing Memory via Binder

Limitations of Binder

Binder - Security

Binder Death Notification

Given a reference to an IBinder object, we can:

Binder Reporting

Binder driver reports various stats on active/failed transactions via /proc/binder/

Replace /proc/binder with /sys/kernel/debug/binder on devices with debugfs enabled.

Additional Binder Resources

Lab: Binder-based Service with AIDL

  1. Start by importing LogBinderDemo application into Eclipse

    1. Menu Bar → File → Import… → Git → Projects from Git → Next >

    2. Under Select Repository Source select URI → Next >

    3. Under Source Git Repository → Location → URI: enter https://github.com/thenewcircle/LogBinderDemo.git → Next >

    4. Under Branch Selection, leave all branches selected (checked) → Next >

    5. Under Local Destination → Destination specify directory of your choice (e.g. ~/android/workspace/LogBinderDemo) → Next >

    6. Under Select a wizard to use for importing projects, leave Wizard for project import as Import existing projects → Next >

    7. Under Import Projects → Projects, leave LogBinderDemo as selected (checked) → Finish

      This project can also be downloaded as a ZIP file

  2. In the Common project:

    1. Create an ILogService.aidl definition that provides the following functionality to clients: 

      public void log(LogMessage logMessage);

    2. You have been given the LogMessage data object that we will pass to our ILogService, but we still need to create the LogMessage.aidl file that notifies AIDL about our custom class.

      At this point, if your project is able to build, you should be able to examine the generated ILogService.java file in the project’s gen/ directory.

  3. In the Service project:

    1. Create an implementation of the Binder service by extending the ILogService.Stub

      Your implementation could simply use android.util.Log.println(int priority, String tag, String msg) to do the logging.

    2. Attach your implementation to the existing wrapper service, allowing clients to discover it.

  4. In the Client project:

    1. Discover and retrieve an ILogService.Stub.Proxy instance with an Intent.

    2. Submit a LogMessage request to the remote ILogService running in a separate process using your Proxy reference.

The solution is provided:

Binder: Review Questions

  1. Why do we need IPC on Android?

  2. What is Binder’s unit of data called?

  3. What is AIDL and when do we use it?

  4. Is the use of AIDL required when consuming or exposing bound services?

  5. Name at least five supported method parameter/return data types that we use in AIDL.

  6. Name at least three types that are not supported.

  7. What is android.os.Bundle?

  8. What is android.os.Parcelable and how do we implement it?

  9. What do we have to do with android.os.Parcelable classes before we can use them in AIDL interfaces?

  10. What is android.os.IBinder?

  11. What is special about file descriptors?

  12. What is the purpose of the directional flag?

  13. What is the purpose of the aidl tool?

  14. What is the purpose of the Stub and the Proxy?

  15. What is the Android library project and when do we use them?

  16. What is the purpose of android.app.Service in the context of bound services?

  17. How do we expose a bound service to other applications?

  18. What is the thread context of an binder service request?

  19. How and when do we bind to non-system services?

  20. What is the life-cycle dependency between bound services and their clients?

  21. What is the purpose of the oneway keyword and when do we use it?

  22. What do we have to worry about when handling call-backs from remote services?

Android Security Essentials

Android Security Architecture

images/AndroidProcessSnapshot-NoPermissions.svg

Application Sandboxing

images/Sandboxing.svg
Sandboxing

Application Signing

images/AppAnatomy.png
Structure of an app

All apps (.apk files) must be digitally signed prior to installation on a device with a certificate whose private key is kept confidential by the developer of the application

Android uses the certificate as a means of:

Identifying the author of an application

Used to ensure the authenticity of future application updates

Establishing trust relationships between applications

Applications signed with the same key (i.e. share the certificate) can share signature-level permissions, user-id (file system resources), and runtime process

Signing Process











Platform Keys

AOSP comes with four platform keys in build/target/product/security/:

Generating Platform Keys

The keys provided by default cannot be used to ship an actual product (enforced by CTS)

To generate our own keys, we can use the supplied development/tools/make_key command for each platform, media, shared, and testkey:

  1. $ rm build/target/product/security/platform.p*

  2. $ SIGNER="/C=US/ST=California/L=San Francisco/O=Marakana Inc./OU=Android/CN=Android Platform Signer/emailAddress=android@marakana.com"

  3. $ echo | development/tools/make_key build/target/product/security/platform "$SIGNER"

  4. Repeat for media, shared, and testkey

Using Platform Keys

To sign our own app using the platform key:

User IDs

File Access

External Storage Volumes

Android provides for devices to include mounted volumes of shared application storage known as "external storage". The name comes from the heritage of Android devices having a physical, removable SD card that was mounted and accessible for all applications to globally read/write data.

Modern devices typically implement emulated external storage in lieu of (or in addition to) a physical storage medium. This is a location on the /data partition that the FUSE daemon mounts and exposes with shared storage permissions. This creates a common API between the services and mounted volumes, regardless of their underlying implementation.

As a rule, external storage volumes must make their contents accessible to a connected host for file transfer via either USB Mass Storage (UMS) or Media Transfer Protocol (MTP). MTP is more commonly used with emulated storage.

The distinction between emulated and physical is separate from that of primary versus secondary external storage (as described previously), which defines the permissions rules for that volume. In CTS testing, Google does not specify which type of storage can be used for primary and/or secondary. Consider the following example storage configuration:

init.hardware.rc
on init
    mkdir /mnt/shell/emulated 0700 shell shell
    mkdir /storage/emulated 0555 root root

    mkdir /mnt/media_rw/sdcard1 0700 media_rw media_rw
    mkdir /storage/sdcard1 0700 root root

    export EXTERNAL_STORAGE /storage/emulated/legacy
    export EMULATED_STORAGE_SOURCE /mnt/shell/emulated
    export EMULATED_STORAGE_TARGET /storage/emulated
    export SECONDARY_STORAGE /storage/sdcard1

    # Support legacy paths
    symlink /storage/emulated/legacy /sdcard
    symlink /storage/emulated/legacy /mnt/sdcard
    symlink /storage/emulated/legacy /storage/sdcard0
    symlink /mnt/shell/emulated/0 /storage/emulated/legacy

…

service sdcard /system/bin/sdcard -u 1023 -g 1023 -l /data/media /mnt/shell/emulated
    class late_start

service fuse_sdcard1 /system/bin/sdcard -u 1023 -g 1023 -w 1023 -d /mnt/media_rw/sdcard1 /storage/sdcard1
    class late_start
    disabled

The EXTERNAL_STORAGE and SECONDARY_STORAGE environment variables configure what volumes Android will use for primary and secondary. This device uses emulated storage for the primary volume, and a physical SD card for secondary external storage.

Android must run a FUSE daemon instance for each active volume, as this is where the permissions are derived. The fuse_sdcard1 instance adds the -w 1023 flag, which is used to enforce write protection on the secondary external storage volume for non-system applications (UID 1023 is the media_rw system-level user).

Multi-User Support

Permissions

Permissions

images/Permissions.svg
Permissions

Google Play Permission Granting

images/GooglePlay-Permissions.png
Facebook required permissions

Using Permissions

Top Ten Bad Permissions (on Google Play)

Using the following permissions will significantly lower the likelihood for an Android app/game to be featured in Google Play (from Google I/O 2012):

  1. android.permission.SEND_SMS and android.permission.RECEIVE_SMS

  2. android.permission.SYSTEM_ALERT_WINDOW

  3. com.android.browser.permission.READ_HISTORY_BOOKMARKS and com.android.browser.permission.WRITE_HISTORY_BOOKMARKS

  4. android.permission.READ_CONTACTS, android.permission.WRITE_CONTACTS, android.permission.READ_CALENDAR, android.permission.WRITE_CALENDAR

  5. android.permission.CALL_PHONE

  6. android.permission.READ_LOGS

  7. android.permission.ACCESS_FINE_LOCATION

  8. android.permission.GET_TASKS

  9. android.permission.RECEIVE_BOOT_COMPLETED

  10. android.permission.CHANGE_WIFI_STATE

Avoid Using Permissions (When You Can)

Permission Enforcement

There are a number of trigger points for security/permission checks:

Kernel / File-system Permission Enforcement

UID-based Permission Enforcement

frameworks/base/cmds/servicemanager/service_manager.c
…
static struct {
    unsigned uid;
    const char *name;
} allowed[] = {
#ifdef LVMX
    { AID_MEDIA, "com.lifevibes.mx.ipc" },
#endif
    { AID_MEDIA, "media.audio_flinger" },
    { AID_MEDIA, "media.player" },
    { AID_MEDIA, "media.camera" },
    { AID_MEDIA, "media.audio_policy" },
    { AID_DRM,   "drm.drmManager" },
    { AID_NFC,   "nfc" },
    { AID_RADIO, "radio.phone" },
    { AID_RADIO, "radio.sms" },
    { AID_RADIO, "radio.phonesubinfo" },
    { AID_RADIO, "radio.simphonebook" },
/* TODO: remove after phone services are updated: */
    { AID_RADIO, "phone" },
    { AID_RADIO, "sip" },
    { AID_RADIO, "isms" },
    { AID_RADIO, "iphonesubinfo" },
    { AID_RADIO, "simphonebook" },
};
…
int svc_can_register(unsigned uid, uint16_t *name)
{
    unsigned n;

    if ((uid == 0) || (uid == AID_SYSTEM))
        return 1;

    for (n = 0; n < sizeof(allowed) / sizeof(allowed[0]); n++)
        if ((uid == allowed[n].uid) && str16eq(name, allowed[n].name))
            return 1;

    return 0;
}
…

Paranoid Network Security

Static Permission Enforcement

Dynamic Permission Enforcement

images/AndroidSecurityCallFlow.svg

Custom Permissions

Before we can enforce our own permissions, we have to declare them using one or more <permission> in

AndroidManifest.xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.marakana.android.myapp" >
  <permission
    android:name="com.example.app.DO_X"
    android:label="@string/do_x_label"
    android:description="@string/do_x_desc"
    android:permissionGroup="android.permission-group.PERSONAL_INFO"
    android:protectionLevel="dangerous" />
  …
</manifest>
About Permission Definition Conflicts

Since Android allows any application to define new permissions they wish to enforce apart from the framework, multiple applications may attempt to define the same permission name (which must be unique) with different parameters.

  • Android operates on a "first definition wins" principle when multiple <permission> elements are found asking for the same name.
    • Only attributes of the first definer (including protectionLevel) are implemented.
    • Android 4.4.3+ gives precedence to system applications when a definition conflict occurs
      • Avoids 3rd party applications pre-empting system applications defining new permissions
  • Android cannot grant permissions that don’t exist yet, and apps only get one chance to receive their permissions grants (at install time).
    • If an application with <uses-permission> is installed before the application containing the corresponding <permission> to define it, the system will not grant the permission to the requesting application until it is updated/reinstalled.
  • When an application defining one or more <permission> elements is uninstalled, those records are removed from the system’s known permissions list.
    • Any applications currently holding that permission will still have the permission granted to them until they are updated/reinstalled.
  • Permission-defining applications can check, via PackageManager.getInstalledPackages() and the PackageInfo.permissions attributes, whether another application has attempted to re-define a <permission> element unexpectedly.

Adding Custom Permissions Dynamically

Permissions by Example

Static Permission Enforcement

Here, we want to restrict access to the com.marakana.android.fibonacciservice.FibonacciService to applications (i.e. clients) that hold USE_FIBONACCI_SERVICE custom permission

  1. We start by by creating a custom permission group (making sure that we name-space it):

    FibonacciService/res/values/strings.xml:
    <?xml version="1.0" encoding="utf-8"?>
<resources>
    …
    <string name="fibonacci_permissions_group_label">Fibonacci Permissions</string>
    …
</resources>
    FibonacciService/AndroidManifest.xml:
    <?xml version="1.0" encoding="utf-8"?>
<manifest …>
    …
    <permission-group
        android:name="com.marakana.android.fibonacciservice.FIBONACCI_PERMISSIONS"
        android:label="@string/fibonacci_permissions_group_label" />
    …
</manifest>
    This permission group is optional - as we could instead use one of the already provided groups

  2. Next, we create a custom permission (again, making sure that we name-space it), while taking advantage of our newly-created permission group:

    FibonacciService/res/values/strings.xml:
    <?xml version="1.0" encoding="utf-8"?>
<resources>
    …
    <string name="use_fibonacci_service_permission_label">use fibonacci service</string>
    <string name="use_fibonacci_service_permission_description">
      applications with this permissions get fibonacci results for free
    </string>
    …
</resources>
    FibonacciService/AndroidManifest.xml:
    <?xml version="1.0" encoding="utf-8"?>
<manifest …>
    …
    <permission-group …/>
    <permission
        android:name="com.marakana.android.fibonacciservice.USE_FIBONACCI_SERVICE"
        android:description="@string/use_fibonacci_service_permission_description"
        android:label="@string/use_fibonacci_service_permission_label"
        android:permissionGroup="com.marakana.android.fibonacciservice.FIBONACCI_PERMISSIONS"
        android:protectionLevel="dangerous" />
    …
</manifest>

  3. Now we can statically require the permission on our FibonacciService service:

    FibonacciService/AndroidManifest.xml:
    <?xml version="1.0" encoding="utf-8"?>
<manifest …>
    …
    <permission-group …/>
    <permission …/>
    <application …>
        <service
            android:name=".FibonacciService"
            android:permission="com.marakana.android.fibonacciservice.USE_FIBONACCI_SERVICE" >
            …
        </service>
    </application>
    …
</manifest>

  4. If we now re-run the FibonacciService and re-run the FibonacciClient, we will notice that the client will fail to launch and adb logcat will show something like: 

    …
W/ActivityManager(   85): Permission Denial: Accessing service ComponentInfo{com.marakana.android.fibonacciservice/com.marakana.android.fibonacciservice.FibonacciService} from pid=540, uid=10043 requires com.marakana.android.fibonacciservice.USE_FIBONACCI_SERVICE
D/AndroidRuntime(  540): Shutting down VM
W/dalvikvm(  540): threadid=1: thread exiting with uncaught exception (group=0x409c01f8)
E/AndroidRuntime(  540): FATAL EXCEPTION: main
E/AndroidRuntime(  540): java.lang.RuntimeException: Unable to resume activity {com.marakana.android.fibonacciclient/com.marakana.android.fibonacciclient.FibonacciActivity}: java.lang.SecurityException: Not allowed to bind to service Intent { act=com.marakana.android.fibonaccicommon.IFibonacciService }
E/AndroidRuntime(  540):    at android.app.ActivityThread.performResumeActivity(ActivityThread.java:2444)
…
E/AndroidRuntime(  540):    at dalvik.system.NativeStart.main(Native Method)
E/AndroidRuntime(  540): Caused by: java.lang.SecurityException: Not allowed to bind to service Intent { act=com.marakana.android.fibonaccicommon.IFibonacciService }
E/AndroidRuntime(  540):    at android.app.ContextImpl.bindService(ContextImpl.java:1135)
E/AndroidRuntime(  540):    at android.content.ContextWrapper.bindService(ContextWrapper.java:370)
E/AndroidRuntime(  540):    at com.marakana.android.fibonacciclient.FibonacciActivity.onResume(FibonacciActivity.java:65)
…
W/ActivityManager(   85):   Force finishing activity com.marakana.android.fibonacciclient/.FibonacciActivity
…

  5. Finally, we can give FibonacciClient a fighting chance by allowing it to use the USE_FIBONACCI_SERVICE permission:

    FibonacciClient/AndroidManifest.xml:
    <?xml version="1.0" encoding="utf-8"?>
<manifest …>
    …
    <uses-permission android:name="com.marakana.android.fibonacciservice.USE_FIBONACCI_SERVICE"/>
    …
</manifest>

  6. We can now observe that our client is again able to use the service

  7. In the Emulator, if we go to Home → Menu → Manage apps → Fibonacci Client → PERMISSIONS, we should see the Fibonacci Permissions group and under it, use fibonacci service permission

Dynamic Permission Enforcement

Here, we want to restrict access to the com.marakana.android.fibonacciservice.IFibonacciServiceImpl's recursive operations (fibJR(long n) and fibNR(long n)) for n > 10 to applications (i.e. clients) that hold USE_SLOW_FIBONACCI_SERVICE custom permission

  1. Like before, we start off by creating a custom permission:

    FibonacciService/res/values/strings.xml:
    <?xml version="1.0" encoding="utf-8"?>
<resources>
    …
    <string name="use_slow_fibonacci_service_permission_label">
        use slow fibonacci service operations
   </string>
    <string name="use_slow_fibonacci_service_permission_description">
        applications with this permissions can melt the CPU and drain the battery
        by using slow fibonacci operations
    </string>
    …
</resources>
    FibonacciService/AndroidManifest.xml:
    <?xml version="1.0" encoding="utf-8"?>
<manifest …>
    …
    <permission-group …/>
    <permission …/>
    <permission
        android:name="com.marakana.android.fibonacciservice.USE_SLOW_FIBONACCI_SERVICE"
        android:description="@string/use_slow_fibonacci_service_permission_description"
        android:label="@string/use_slow_fibonacci_service_permission_label"
        android:permissionGroup="com.marakana.android.fibonacciservice.FIBONACCI_PERMISSIONS"
        android:protectionLevel="dangerous" />
    …
</manifest>

  2. Next, we update our IFibonacciServiceImpl to enforce this permission dynamically - via the android.content.Context that we receive through the constructor:

    FibonacciService/src/com/marakana/android/fibonacciservice/IFibonacciServiceImpl.java:
    package com.marakana.android.fibonacciservice;

import android.content.Context;
…
public class IFibonacciServiceImpl extends IFibonacciService.Stub {
    …
    private final Context context;

    public IFibonacciServiceImpl(Context context) {
        this.context = context;
    }

    private long checkN(long n) {
        if (n > 10) {
            this.context.enforceCallingOrSelfPermission(
                    Manifest.permission.USE_SLOW_FIBONACCI_SERVICE, "Go away!");
        }
        return n;
    }
    …
    public long fibJR(long n) {
        …
        return FibLib.fibJR(this.checkN(n));
    }
    …
    public long fibNR(long n) {
        …
        return FibLib.fibNR(this.checkN(n));
    }
    …
}

  3. We have to update FibonacciService to invoke the new IFibonacciServiceImpl's constructor:

    FibonacciService/src/com/marakana/android/fibonacciservice/FibonacciService.java:
    …
public class FibonacciService extends Service {
    …
    @Override
    public void onCreate() {
        …
        this.service = new IFibonacciServiceImpl(super.getApplicationContext());
        …
    }
    …
}

  4. If we now re-run the FibonacciService and re-run the FibonacciClient for a recursive operation with n > 10, we will notice that the client will fail and adb logcat will show something like: 

    …
D/IFibonacciServiceImpl(  617): fib(15, RECURSIVE_NATIVE)
D/IFibonacciServiceImpl(  617): fibNR(15)
W/dalvikvm(  604): threadid=11: thread exiting with uncaught exception (group=0x409c01f8)
E/AndroidRuntime(  604): FATAL EXCEPTION: AsyncTask #1
E/AndroidRuntime(  604): java.lang.RuntimeException: An error occured while executing doInBackground()
…
E/AndroidRuntime(  604):    at java.lang.Thread.run(Thread.java:856)
E/AndroidRuntime(  604): Caused by: java.lang.SecurityException: Go away!: Neither user 10043 nor current process has com.marakana.android.fibonacciservice.USE_SLOW_FIBONACCI_SERVICE.
…

  5. Finally, we can allow FibonacciClient to melt our CPU and drain our battery by allowing it to use the USE_SLOW_FIBONACCI_SERVICE permission:

    FibonacciClient/AndroidManifest.xml:
    <?xml version="1.0" encoding="utf-8"?>
<manifest …>
    …
    <uses-permission android:name="com.marakana.android.fibonacciservice.USE_SLOW_FIBONACCI_SERVICE"/>
    …
</manifest>

  6. We can now observe that our client is again able to use recursive fibonacci operations even for n > 10

  7. In the Emulator, if we go to Home → Menu → Manage apps → Fibonacci Client → PERMISSIONS → Fibonacci Permissions, we should see both use fibonacci service and use slow fibonacci service operations permissions

Lab: Custom Permissions

ContentProvider URI Permissions

Public vs. Private Components

Intent Broadcast Permissions

Pending Intents

Review Questions

  1. What is the basic philosophy of the Android security model?

  2. What does Android do to implement this philosophy?

  3. How and when are permissions used, granted, and enforced?

  4. What is a confused deputy attack?

  5. What is a collusion attack?

  6. Which part of Android is ultimately responsible for providing the system security?

  7. How does native code differ from Java code when it comes to its security context?

  8. Why do we sign applications?

  9. What are the four platform keys?

  10. What is the relationship between apps and Linux user IDs?

  11. When are the user IDs assigned to apps?

  12. Where are the user ID assignments recorded?

  13. What is the relationship between application user IDs and its file-system resources?

  14. Where is the home of an app with the package name com.foo.bar?

  15. How do apps share a user IDs?

  16. How do apps share a process? Provide at least one use-case for this.

  17. Name at least five (file) system volumes (think mount points).

  18. What is the purpose of /data/dalvik-cache/?

  19. What is the purpose of run-as command?

  20. How do you use permissions?

  21. What happens if you don’t use a permission but you attempt to use a restricted API?

  22. How are the permissions enforced?

  23. How do bound services enforce their permissions? What makes that possible?

  24. What are the attributes of the <permission … /> tag?

  25. What are the possible values for protectionLevel attribute?

  26. What does protectedLevel=signatureOrSystem mean?

  27. What is the value of grouping permissions?

  28. What is the significance of android:grantUriPermission attribute on content providers?

  29. What is the significance android:exported attribute on Android components?

  30. How do you protected unauthorized applications from receiving your broadcasts (intents)?

  31. What’s the worry with pending intents?

Building Android From Source

Why Build Android From Source?

Setting up the Build Environment

Downloading the Source Tree

Android Source Code Structure

Android Build System

Initializing the Build Environment

Choosing the Build Target

Additional CodeName-BuildType combinations may be available based on what build/envsetup.sh finds (usually in the device/ folder)
Building for real hardware usually requires that we get proprietary binaries (mostly user-space HAL). For Nexus and other Google-supported devices, we can go to http://code.google.com/android/nexus/drivers.html, which allows us to download scripts, which in turn extract binaries from the connected devices (via adb pull) into vendor/ directory tree structure.

Compiling Android

Makefile targets

Android build system Makefile (actually build/core/main.mk) includes some of the following targets:

Make targets Description

droid

The default target (build the full system)

clean

Equivalent to rm -rf out/ (same as make clobber)

installclean

Deletes all of the files that change between different build types, like make user vs. make sdk

dataclean

Delete files in the staging and emulator data partitions: data/*, data-qemu/*, and userdata-qemu.img

snod

Quickly rebuild the system image from built packages

offline-sdk-docs

Generate the HTML for the developer SDK docs

doc-comment-check-docs

Check HTML doc links and validity, without generating HTML

libandroid_runtime

All the JNI framework stuff

framework

All the java framework stuff

services

The system server (Java) and friends

sdk

Build the Android SDK (tools)

help

Display a help message listing some of these targets

modules

Display a list of modules that can be built (where each module name is specified by LOCAL_MODULE)

<module-name>

Make just a specific module (same as cd module/dir && mm)

clean-<module-name>

Clean just a specific module

otacerts

OTA keys that are used to verify OTA packages

…

Etc.

Examining the Built Images

Running Custom Android Build on Emulator

Running Custom Android Build on Real Hardware

Building the Linux Kernel

It is easiest to build the Linux kernel on a Linux OS. While other host OSs can also be used, they are not trivial to setup.

Getting the Kernel

$ git clone https://android.googlesource.com/kernel/common.git
$ git clone https://android.googlesource.com/kernel/goldfish.git
$ git clone https://android.googlesource.com/kernel/msm.git
$ git clone https://android.googlesource.com/kernel/omap.git
$ git clone https://android.googlesource.com/kernel/samsung.git
$ git clone https://android.googlesource.com/kernel/tegra.git

Building Kernel for the Emulator (Goldfish)

  1. Start the emulator

  2. Get the existing kernel version from /proc/version (since uname does not exist on Android) 

    $ adb shell cat /proc/version
Linux version 3.4.0-gd853d22 (nnk@nnk.mtv.corp.google.com) (gcc version 4.6.x-google 20120106 (prerelease) (GCC) ) #1 PREEMPT Tue Jul 9 17:46:46 PDT 2013

  3. Get the corresponding kernel version (here, we are getting the kernel for goldfish, the emulator) 

    $ git clone https://android.googlesource.com/kernel/goldfish.git
$ cd goldfish/
$ git branch -a
$ git checkout -t remotes/origin/android-goldfish-3.4
    Alternatively, we could directly clone the goldfish 3.4 branch
    $ git clone https://android.googlesource.com/kernel/goldfish.git -b android-goldfish-3.4

  4. Specify the target architecture and cross compiler 

    $ export ARCH=arm
$ export CROSS_COMPILE=$ANDROID_EABI_TOOLCHAIN/arm-linux-androideabi-

    When compiling for x86, set the following instead:

    $ export ARCH=x86
$ export REAL_CROSS_COMPILE=$ANDROID_EABI_TOOLCHAIN/i686-linux-android-
$ export CROSS_COMPILE=$AOSP/external/qemu/distrib/kernel-toolchain/android-kernel-toolchain-

    Android’s x86 compiler is NDK-compatible, so it enforces -mfpmath=sse and -fpic by default. When building the kernel, we need to disable this.

    In this case, $CROSS_COMPILE points to a toolchain of shell scripts, which add -mfpmath=387 -fno-pic flags before calling into $REAL_CROSS_COMPILE toolchain.
    Android’s own build system uses ARCH and CROSS_COMPILE env vars, so these need to be cleared before building AOSP.

    As an alternative, these can be also passed directly to the make commands: $ make … ARCH=… CROSS_COMPILE=…

About Cross Compilers

The environment variable ANDROID_EABI_TOOLCHAIN was initialized by lunch. Its value varies based on the version of Android, host architecture, and target architecture:

  • Android 4.1 (JB)
    • MacOS X
      • arm: $AOSP_HOME/prebuilts/gcc/darwin-x86/arm/arm-linux-androideabi-4.6/bin
      • x86: $AOSP_HOME/prebuilts/gcc/darwin-x86/x86/i686-android-linux-4.4.3/bin
    • Linux
      • arm: $AOSP_HOME/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.6/bin
      • x86: $AOSP_HOME/prebuilts/gcc/linux-x86/x86/i686-android-linux-4.4.3/bin
  • Android 4.4 (KK)
    • MacOS X
      • arm: $AOSP_HOME/prebuilts/gcc/darwin-x86/arm/arm-linux-androideabi-4.7/bin
      • x86: $AOSP_HOME/prebuilts/gcc/darwin-x86/x86/i686-linux-android-4.7/bin
    • Linux
      • arm: $AOSP_HOME/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.7/bin
      • x86: $AOSP_HOME/prebuilts/gcc/linux-x86/x86/i686-linux-android-4.7/bin

  1. Generate the kernel configuration using default config supplied with the source tree. This will write a .config file into the directory 

    $ make goldfish_armv7_defconfig

    When compiling for ARMv5 or x86, run the following instead:

    $ make goldfish_defconfig

    As an alternative, we could get an existing kernel configuration file (with all of the Android/emulator specific options) by pulling it from a running emulator:

    $ adb pull /proc/config.gz .
$ gunzip config.gz
$ mv config .config

    You could also optionally take a look at the existing configuration options and make changes as desired

    $ make menuconfig ARCH=arm
    • For example, you could set your General setup → Local version - append to kernel release to something like -marakana-example

  2. Now we are ready to compile 

    $ make
    We’ve had issues compiling the kernel on Mac OS X Lion (10.7).

  3. The resulting kernel will be compressed to arch/arm/boot/zImage

  4. Run the emulator with our new kernel 

    $ $ANDROID_HOST_OUT/bin/emulator -kernel /path/to/common/arch/arm/boot/zImage

  5. And the result is

    screens/CustomKernel.png

Review Questions

  1. What is repo?

  2. What Android’s build system based on?

  3. What is the first thing you need to do before you can build Android (after you get the source)?

  4. What is the unit of build?

  5. What are the different build types and how do they differ?

  6. What is the purpose of ccache?

  7. What is the end result of a build?

  8. What’s inside recovery.img?

  9. What is the purpose of fastboot?

  10. What is the purpose of CROSS_COMPILE variable and what do you set it to when compiling the kernel?

Android Startup

images/AndroidProcessSnapshot.svg

Bootloading the Kernel

  1. On power-up, CPU is uninitialized - wait for stable power

  2. Execute Boot ROM (hardwired into CPU)

    1. Locate the first-stage boot loader

    2. Load the first-stage boot loader into internal RAM

    3. Jump to first-stage boot loader’s memory location to execute it

  3. First-stage boot loader runs

    1. Detect and initialize external RAM

    2. Locate the second-stage boot loader

    3. Load the second-stage boot loader into external RAM

    4. Jump to the second-stage boot loader’s memory location to execute it

  4. Second-stage boot loader runs

    1. Setup file systems (typically on Flash media)

    2. Optionally setup display, network, additional memory, and other devices

    3. Enable additional CPU features

    4. Enable low-level memory protection

    5. Optionally load security protections (e.g. ROM validation code)

    6. Locate Linux Kernel

    7. Load Linux Kernel into RAM

    8. Place Linux Kernel boot parameters into memory so that kernel knows what to run upon startup

    9. Jump to Linux Kernel memory address to run it

  5. Linux Kernel runs

    1. Build a table in RAM describing the layout of the physical memory

    2. Initialize and setup input devices

    3. Initialize and setup disk (typically MTD) controllers and map available block devices in RAM

    4. Initialize Advanced Power Management (APM) support

    5. Initialize interrupt handlers: Interrupt Descriptor Table (IDT), Global Descriptor Table (GDT), and Programmable Interrupt Controllers (PIC)

    6. Reset the floating-point unit (FPU)

    7. Switch from real to protected mode (i.e. enable memory protection)

    8. Initialize segmentation registers and a provisional stack

    9. Zero uninitialized memory

    10. Decompress the kernel image

    11. Initialize provisional kernel page tables and enable paging

    12. Setup kernel mode stack for process 0

    13. Fill the IDT with null interrupt handlers

    14. Initialize the first page frame with system parameters

    15. Identify the CPU model

    16. Initialize registers with the addresses of the GDT and IDT

    17. Initialize and start the kernel

      1. Scheduler

      2. Memory zones

      3. Buddy system allocator

      4. IDT

      5. SoftIRQs

      6. Date and Time

      7. Slab allocator

      8. …

    18. Create process 1 (/init) and run it

Android’s init Startup

images/StartupWalkthru.svg

SurfaceFlinger startup

  1. SurfaceFlinger starts from /init.rc 

    service surfaceflinger /system/bin/surfaceflinger
    class main
    user system
    group graphics drmrpc
    onrestart restart zygote

  2. This translates to frameworks/native/services/surfaceflinger/main_surfaceflinger.cpp:main()

  3. The daemon registers a frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp instance with frameworks/native/libs/binder/IServiceManager.cpp:addService(), which effectively talks to servicemanager daemon previously started by init

Mediaserver Startup

  1. Mediaserver starts from /init.rc 

    service media /system/bin/mediaserver
    class main
    user media
    group audio camera inet net_bt net_bt_admin net_bw_acct drmrpc mediadrm
    ioprio rt 4

  2. This translates to frameworks/av/media/main_mediaserver.cpp:main()

  3. The command process initializes all of the native media system service components via their instantiate() method, which creates the object and registers it with frameworks/native/libs/binder/IServiceManager.cpp:addService():

    1. frameworks/av/services/AudioFlinger.cpp

    2. frameworks/av/media/libmediaplayerservice/MediaPlayerService.cpp

    3. frameworks/av/services/camera/libcameraservice/CameraService.cpp

    4. frameworks/av/services/AudioPolicyService.cpp

Zygote Startup

  1. Zygote starts from /init.rc 

    service zygote /system/bin/app_process -Xzygote /system/bin --zygote --start-system-server
    socket zygote stream 666
    onrestart write /sys/android_power/request_state wake
    onrestart write /sys/power/state on
    onrestart restart media
    onrestart restart netd

  2. This translates to frameworks/base/cmds/app_process/app_main.cpp:main()

  3. The command app_process then launches frameworks/base/core/java/com/android/internal/os/ZygoteInit.java:main() inside the runtime

  4. ZygoteInit.main() then

    1. Registers for zygote socket

    2. Pre-loads classes defined in frameworks/base/preloaded-classes (1800+)

    3. Pre-loads resources preloaded_drawables and preloaded_color_state_lists from frameworks/base/core/res/res/values/arrays.xml

    4. Runs garbage collector (to clean the memory as much as possible)

    5. Forks itself to start system_server

    6. Starts listening for requests to fork itself for other apps

System Server Startup

  1. When Zygote forks itself to launch the system_server process (in ZygoteInit.java:startSystemServer()), it executes frameworks/base/services/java/com/android/server/SystemServer.java:main()

  2. The SystemServer.java:main() method loads android_servers JNI lib from frameworks/base/services/jni and invokes nativeInit() native method

  3. Before nativeInit() runs, the JNI loader first runs frameworks/base/services/jni/onload.cpp:JNI_OnLoad(), which registers native services - to be used as JNI counterparts to Java-based service manager loaded later

  4. Now frameworks/base/services/jni/com_android_server_SystemServer.cpp:nativeInit() is invoked, instantiating the following native components before starting the Java services:

    1. frameworks/native/services/sensorservice/SensorService.cpp

  5. The SystemServer.java:main() method then starts Java service managers in a ServerThread instance (via the initAndLoop() method), readies them, and registers each one with frameworks/base/core/java/android/os/ServiceManager:addService() (which in turn delegates to to ServiceManagerNative.java, which effectively talks to servicemanager daemon previously started by init):

    1. frameworks/base/services/java/com/android/server/PowerManagerService.java

    2. frameworks/base/services/java/com/android/server/am/ActivityManagerService.java

    3. frameworks/base/services/java/com/android/server/TelephonyRegistry.java

    4. frameworks/base/services/java/com/android/server/PackageManagerService.java

    5. frameworks/base/services/java/com/android/server/BatteryService.java

    6. frameworks/base/services/java/com/android/server/VibratorService.java

    7. etc

  6. Execution transfers to frameworks/base/services/java/com/android/server/am/ActivityManagerService.java:systemReady() to complete the boot process via:

    1. a broadcast intent with android.intent.action.PRE_BOOT_COMPLETED action (to give apps a chance to reach to boot upgrades)

    2. an activity intent with android.intent.category.HOME category to launch the Home (or Launcher) application

      1. frameworks/base/services/java/com/android/server/am/ActivityStackSupervisor.java:resumeTopActivitiesLocked()

      2. frameworks/base/services/java/com/android/server/am/ActivityStack.java:resumeTopActivityLocked()

      3. frameworks/base/services/java/com/android/server/am/ActivityStackSupervisor.resumeHomeActivity()

      4. frameworks/base/services/java/com/android/server/am/ActivityManagerService.java:startHomeActivityLocked()

  7. With the Home application launched, ActivityStackSupervisor.java:ActivityStackSupervisorHandler receives an IDLE_NOW_MSG event, triggering ActivityStackSupervisor.java:activityIdle()

  8. Finally, this triggers ActivityManagerService.java:finishBooting() to complete the boot process, as it:

    1. sets the system property sys.boot_completed=1

    2. sends out a broadcast intent with android.intent.action.BOOT_COMPLETED action, which launches applications subscribed to this intent (while using android.permission.RECEIVE_BOOT_COMPLETED)

Review Questions

  1. What is the purpose of the second-stage bootloader?

  2. What is the first thing that runs (in the user-space) after the kernel is initialized?

  3. Where does this process (with PID==1) get its initialization instructions from?

  4. What are actions (of process with PID==1)?

  5. Name at least three triggers and three commands for actions.

  6. What are services (of process with PID==1)?

  7. How are services launched?

  8. What’s the difference between critical and non-critical services?

  9. Name at least five services started by process with PID==1.

  10. What is the purpose of ueventd process?

  11. What is the purpose of zygote process and what does it do upon initialization?

  12. What is the purpose of system_server process and what does it do upon initialization?

  13. Who is responsible for launching the Home application and how is it done?

  14. How do applications that want to start on boot get launched?

Android Subsystems

The objective of this module is to explain the inter-workings of various Android subsystems. There are close to sixty various services in ICS release. In this module, we’ve hand-picked some of the more common ones. By the end of the module, you should start to understand some common traits of Android subsystem architectures, such as use of the Binder for inter-process communication, and use of JNI for Java-C interaction.

Android services are the key to exposing lower level functionality of the hardware and the Linux kernel to the high level Android apps. Understanding how they work creates the opportunity to customize and extend their behavior, or add another service altogether.

$ adb shell service list
Found 56 services:
0       phone: [com.android.internal.telephony.ITelephony]
1       iphonesubinfo: [com.android.internal.telephony.IPhoneSubInfo]
2       simphonebook: [com.android.internal.telephony.IIccPhoneBook]
3       isms: [com.android.internal.telephony.ISms]
4       samplingprofiler: []
5       diskstats: []
6       appwidget: [com.android.internal.appwidget.IAppWidgetService]
7       backup: [android.app.backup.IBackupManager]
8       uimode: [android.app.IUiModeManager]
9       usb: [android.hardware.usb.IUsbManager]
10      audio: [android.media.IAudioService]
11      wallpaper: [android.app.IWallpaperManager]
12      dropbox: [com.android.internal.os.IDropBoxManagerService]
13      search: [android.app.ISearchManager]
14      country_detector: [android.location.ICountryDetector]
15      location: [android.location.ILocationManager]
16      devicestoragemonitor: []
17      notification: [android.app.INotificationManager]
18      mount: [IMountService]
19      throttle: [android.net.IThrottleManager]
20      connectivity: [android.net.IConnectivityManager]
21      wifi: [android.net.wifi.IWifiManager]
22      wifip2p: [android.net.wifi.p2p.IWifiP2pManager]
23      netpolicy: [android.net.INetworkPolicyManager]
24      netstats: [android.net.INetworkStatsService]
25      textservices: [com.android.internal.textservice.ITextServicesManager]
26      network_management: [android.os.INetworkManagementService]
27      clipboard: [android.content.IClipboard]
28      statusbar: [com.android.internal.statusbar.IStatusBarService]
29      device_policy: [android.app.admin.IDevicePolicyManager]
30      accessibility: [android.view.accessibility.IAccessibilityManager]
31      input_method: [com.android.internal.view.IInputMethodManager]
32      window: [android.view.IWindowManager]
33      alarm: [android.app.IAlarmManager]
34      vibrator: [android.os.IVibratorService]
35      battery: []
36      hardware: [android.os.IHardwareService]
37      content: [android.content.IContentService]
38      account: [android.accounts.IAccountManager]
39      permission: [android.os.IPermissionController]
40      cpuinfo: []
41      gfxinfo: []
42      meminfo: []
43      activity: [android.app.IActivityManager]
44      package: [android.content.pm.IPackageManager]
45      telephony.registry: [com.android.internal.telephony.ITelephonyRegistry]
46      usagestats: [com.android.internal.app.IUsageStats]
47      batteryinfo: [com.android.internal.app.IBatteryStats]
48      power: [android.os.IPowerManager]
49      entropy: []
50      sensorservice: [android.gui.SensorServer]
51      media.audio_policy: [android.media.IAudioPolicyService]
52      media.camera: [android.hardware.IAudioPolicyServiceService]
53      media.player: [android.media.IMediaPlayerService]
54      media.audio_flinger: [android.media.IAudioFlinger]
55      SurfaceFlinger: [android.ui.ISurfaceComposer]

Vibrator on Android

images/VibratorArchitecture.svg
Vibrator Service Architecture

Power on Android

images/PowerManagerServiceArchitecture.svg
Power Service Architecture

Resources

Alarm on Android

images/AlarmManagerServiceArchitecture.svg
Alarm Service Architecture

Package Management on Android

images/PackageManagerServiceArchitecture.svg
Package Manager Service Architecture

WiFi on Android

Wifi Service exposes WiFi functionality of the underlying system to the application layer via WifiManager class. The key differentiation between wifi stack and some other ones is that the wifi stack primarily uses the wpa_supplicant to talk to the Wifi driver.

images/WifiArchitecture.svg
Wifi Service Architecture

Resources

Location on Android

images/LocationManagerServiceArchitecture.svg
Location Service Architecture

Audio on Android

images/AudioPlaybackServiceArchitecture.svg
Audio Architecture (playing audio)

images/AudioManagerServiceArchitecture.svg

Resources

Media on Android

Introduction

The media framework are the APIs and libraries used for controlling playback and recording of video/audio. Since everything involving video and audio require a lot of computing power, mobile devices usually use a lot of special-purpose hardware for this, compared to desktop computers where there normally is enough raw power to run most/everything in software. Full-software solutions normally are more flexible and portable, but obviously require more processor power (which also might give a higher power consumption).

Normally only the highest level APIs are public, to allow for flexibility in the implementation internally.

Typical stack of function calls

  1. java: android.media.MediaPlayer
    frameworks/base/media/java/android/media/MediaPlayer.java

  2. JNI: frameworks/base/media/jni/android_media_MediaPlayer.cpp
    Quite straight mapping of java native functions to the C++ MediaPlayer class

  3. C++: MediaPlayer
    frameworks/av/media/libmedia/mediaplayer.cpp

  4. IMediaPlayer, IMediaPlayerService
    frameworks/av/include/media/IMediaPlayer.h
    frameworks/av/include/media/IMediaPlayerService.h

  5. BpMediaPlayer: Binder client/proxy interface
    frameworks/av/media/libmedia/IMediaPlayer.cpp

  6. BnMediaPlayer: Binder implementation interface (running in the media server)

  7. BnMediaPlayer implemented by MediaPlayerService::Client
    frameworks/av/libmediaplayservice/MediaPlayerService.cpp

  8. MediaPlayerService backed by different implementations:

    • PVPlayer (old OpenCORE based player, phased out in gingerbread, removed later)
    • StagefrightPlayer
      frameworks/av/media/libmediaplayerservice/StagefrightPlayer.cpp
      • AwesomePlayer
        frameworks/av/media/libstagefright/AwesomePlayer.cpp
      • OMXCodec
        frameworks/av/media/libstagefright/OMXCodec.cpp
        Pull based stagefright element, wrapping IOMX
      • AudioPlayer
        frameworks/av/media/libstagefright/AudioPlayer.cpp
    • NuPlayer (new in ICS/Honeycomb only for HTTP Live Streaming)
      frameworks/av/media/libmediaplayerservice/nuplayer/NuPlayerDriver.cpp
      • ACodec
        frameworks/av/media/libstagefright/ACodec.cpp message passing based frontend on top of IOMX

  9. MediaPlayerService::AudioOutput
    frameworks/av/libmediaplayservice/MediaPlayerService.cpp

  10. AudioTrack
    frameworks/av/media/libmedia/AudioTrack.cpp

  11. IAudioTrack

    • BpAudioTrack, BnAudioTrack

  12. AudioFlinger::TrackHandle
    frameworks/av/services/audioflinger/AudioFlinger.cpp

The actual decoding/encoding in the OMXCodec and ACodec class is implemented via the IOMX interface:

The applications may also use OpenSL ES for playing back audio this internally uses OMXCodec from stagefright for decoding of the audio. (Code for this is in system/media/opensles/libopensles/android_SfPlayer.cpp in gingerbread, in system/media/wilhelm/src/android/android_AudioSfDecoder.cpp in ICS.) In ICS, there’s also a an OpenMAX AL API that can play back video, using the IMediaPlayerService interface.

The media server process

Most of the APIs above in section 2 can either run in the calling application processes, or in the media server. Each of the Binder APIs (IAPI/BpAPI/BnAPI, such as IOMX, BpOMX, BnOMX) can proxy calls into the media server. The caller only gets an instance of the generic interface, e.g. IMediaPlayerm and does not know whether this is a proxy for calling the same interface in another process (BpOMX) or the actual implementation running within the same process (BnOMX).

Thanks to this, the whole concept of some parts of the API running within a separate process is mostly transparent when working with the APIs.

Stagefright

Intro

Stagefright is the library containing most of the implementation of the media framework. It first appeared in Eclair, was made default player for all video (except for RTSP streaming which was still handled by OpenCORE) in Froyo, and in Gingerbread it had gained RTSP streaming support and was mature enough to be used for video recording, too, so OpenCORE was removed in Gingerbread. Since then, it has still evolved a bit.

Stagefright is mainly a generic pipeline kind of framework, similar to GStreamer, but much smaller and simpler. (It is purpose-written from scratch to fill the exact needs of Android, while OpenCORE was an already existing framework that PacketVideo provided at the time. Stagefright is a few orders of magnitude smaller than OpenCORE while still fulfilling all the needs Android has.)

GStreamer has the same element/pipeline design, but is much more flexible supporting a number of modes of operation and a lot more elements, while Stagefright is simpler and smaller, since it only does exactly what is needed in Android.

Pipeline

The pipeline in Stagefright is pull-based, meaning that each element deriving from the base class MediaSource implements a method read(), which when called will block until the MediaSource element has produced one MediaBuffer which is returned. When the MediaSource subclass object is created, it is normally given an upstream MediaSource object to read from, e.g. for OMXCodec, the MediaSource is given as parameter to OMXCodec::Create().

images/MediaFrameworkPipeline.svg
An example showing the pipeline decoding audio and video from an MP4 file.
Arrows indicate direction of read() requests as data flows through the pipeline.

For OMXCodec, the read() method internally reads a packet of compressed data from the source (which might e.g. be MPEG4Source within MPEG4Extractor in the case above), passes the packet to the actual OpenMAX codec for decoding, and when the decoded raw audio data is available, it is returned by the read() method.

Similarly, the MPEG4Source object will block in the read() method until it has read one packet from the source file, which is then returned to the caller. With this design, elements can be more or less arbitrarily connected, as long as one element can handle the data type that the input element produces.

Stagefright contains the following kinds of pipeline element implementations:

Players
images/MediaFrameworkFactories.svg
Media players register with the framework via factories.
The system registers core factories via MediaPlayerFactory::registerBuiltinFactories() on intialization of the MediaPlayerService.

OpenMAX IL 5.1 Overview

OpenMAX IL in Stagefright

Stagefright provides wrapping around OMX core implementations. The main client api is called IOMX (frameworks/av/include/media/IOMX.h), which is a Binder interface with a proxy (BpOMX) and backend (BnOMX), allowing the caller and implementation to reside in separate processes. BnOMX is implemented by the OMX class in frameworks/av/media/libstagefright/omx/OMX.cpp, which uses the OMXMaster class in frameworks/av/media/libstagefright/omx/OMXMaster.cpp for querying and instantiating components from a number of different implementation sources. OMXMaster loads one or more plugins (OMXPluginBase, defined in frameworks/native/include/media/hardware/OMXPluginBase.h) that are analogous to normal OMX IL cores.

OMXMaster loads a shared library named libstagefrighthw.so, which should have a function named ZN7android15createOMXPluginEv(android::createOMXPlugin()) or createOMXPlugin (since ICS, both are tried, earlier, only the former was tried), that returns an OMXPluginBase pointer when called.

This library, libstagefrighthw.so, is vendor specific. Examples of implementations of this library, wrapping a normal OMX IL core, are available in:

All of these are very similar they simply load an OMX core from a shared library, load the OMX core function pointers from the library, and map the OMXPluginBase methods to the OMX core.

For software codecs, there’s no full proper OMX IL core in ICS, all decoders are mapped straight from the OMXPluginBase, implemented in frameworks/av/media/libstagefright/omx/SoftOMXPlugin.cpp. All software encoders (and in gingerbread, all software decoders too) are hooked up directly from the OMXCodec class, where the direct constructors of the software encoder classes are called, without any indirection via OMX like interfaces these software codecs implement the MediaSource interface directly. Due to this, these codecs cannot be accessed via the IOMX layer.

The development seems to be moving towards the OMX interfaces namely, in ICS, the software decoders have been converted to use OMX. The new ACodec class in ICS only uses codecs via the OMX interfaces.

Adding a custom OpenMAX IL plugin

Implement an OMX core for the codecs

Examples:

The OMX core acts as a registry for the available codecs. If the build configuration already contains a vendor specific OMX core, the new OMX components can either be added to this OMX core, or a new separate OMX core can be added.

Build libstagefrighthw

Build libstagefright, containing an OMXPluginBase implementation that loads the OMX core. Examples:

If one chooses to add a second OMX core, a second libstagefrighthw (with a slightly different name) has to be created, and this also requires modifications to frameworks/av/media/libstagefright/omx/OMXMaster.cpp, in order to be able to handle two different vendor libraries at the same time. Therefore, it’s probably best to integrate new components within the current vendor OMX core if one already exists.

Implement OMX components for the codecs

This is the actual wrapping of the codecs. Simple examples (which don’t use the proper real OMX API but only stagefright’s own internal API) are available in

  1. frameworks/av/media/libstagefright/omx/SoftOMXComponent.cpp

  2. frameworks/av/media/libstagefright/omx/SimpleSoftOMXComponent.cpp

  3. frameworks/av/media/libstagefright/codecs/aacdec/SoftAAC.cpp

These contain all the logic any OMX component needs to have, but the external API isn’t the proper public OMX version, but only stagefright-internal ones. Real implementations with the proper public OMX API are available in e.g.:

Register the new OMX components

Register the new OMX components by adding entries for them to the /system/etc/media_codecs.xml file on your system image. Stagefright parses this file to determine which components (if any) support a given MIME type. The default implementation in AOSP includes the software codecs provided by Google for encode and decode.

Google providing these implementations does not inherently grant license to use them in a product. It is up to OEMs to secure all necessary licenses for the codecs they provide on the end devices.
device/generic/goldfish/camera/mdeia_codecs.xml
<MediaCodecs>
    <Decoders>
        <MediaCodec name="OMX.google.mp3.decoder" type="audio/mpeg" />
        <MediaCodec name="OMX.google.amrnb.decoder" type="audio/3gpp" />
        <MediaCodec name="OMX.google.amrwb.decoder" type="audio/amr-wb" />
        <MediaCodec name="OMX.google.aac.decoder" type="audio/mp4a-latm" />
        <MediaCodec name="OMX.google.g711.alaw.decoder" type="audio/g711-alaw" />
        <MediaCodec name="OMX.google.g711.mlaw.decoder" type="audio/g711-mlaw" />
        <MediaCodec name="OMX.google.vorbis.decoder" type="audio/vorbis" />
        <MediaCodec name="OMX.google.gsm.decoder" type="audio/gsm" />
        <MediaCodec name="OMX.google.mpeg4.decoder" type="video/mp4v-es" />
        <MediaCodec name="OMX.google.h263.decoder" type="video/3gpp" />
        <MediaCodec name="OMX.google.h264.decoder" type="video/avc" />
        <MediaCodec name="OMX.google.vp8.decoder" type="video/x-vnd.on2.vp8" />
        <MediaCodec name="OMX.google.vp9.decoder" type="video/x-vnd.on2.vp9" />
    </Decoders>
    <Encoders>
        <MediaCodec name="OMX.google.aac.encoder" type="audio/mp4a-latm" />
        <MediaCodec name="OMX.google.amrnb.encoder" type="audio/3gpp" />
        <MediaCodec name="OMX.google.amrwb.encoder" type="audio/amr-wb" />
        <MediaCodec name="OMX.google.h263.encoder" type="video/3gpp" />
        <MediaCodec name="OMX.google.h264.encoder" type="video/avc" />
        <MediaCodec name="OMX.google.mpeg4.encoder" type="video/mp4v-es" />
        <MediaCodec name="OMX.google.flac.encoder" type="audio/flac" />
        <MediaCodec name="OMX.google.vp8.encoder" type="video/x-vnd.on2.vp8" />
    </Encoders>
</MediaCodecs>
Resources

Telephony on Android

In this section, we’ll explore the inter-workings of the telephony stack. We’ll start with the standard Android Phone app, and trace the execution of placing a call all the way down to RIL daemon.

Telephony Manager

Android Framework provides the Telephony Manager class in android.telephony package. The Telephony Manager allows you to monitor the state of the mobile network connection. However, Telephony Manager does not allow you to place or manage any calls. Only the Phone app can initiate and answer phone calls.

Phone App

Some of this content comes from the actual AOSP source code, licensed under Apache 2 License.
images/TelephonyArchitecture.svg
Android Telephony Architecture Diagram
Overview

Phone app does come with some UI components, such as the dialer, but the most significant part is its handling of CALL and CALL_PRIVILEGED intents to do the actual dialing of a number.

OutgoingCallBroadcaster

OutgoingCallBroadcaster activity receives CALL and CALL_PRIVILEGED Intents, and broadcasts the ACTION_NEW_OUTGOING_CALL intent which allows other applications to monitor, redirect, or prevent the outgoing call. After the other applications have had a chance to see the ACTION_NEW_OUTGOING_CALL intent, it finally reaches the OutgoingCallReceiver, which passes the (possibly modified) intent on to the SipCallOptionHandler, which will ultimately start the call using the CallController.placeCall() API.

CallController

CallController handler is the phone app module in charge of call control. This is a singleton object which acts as the interface to the telephony layer (and other parts of the Android framework) for all user-initiated telephony functionality, like making outgoing calls.

This functionality includes things like:

The single CallController instance stays around forever; it’s not tied to the lifecycle of any particular Activity (like the InCallScreen). There’s also no implementation of onscreen UI here (that’s all in InCallScreen).

Note that this class does not handle asynchronous events from the telephony layer, like reacting to an incoming call; see CallNotifier for that. This class purely handles actions initiated by the user, like outgoing calls.

placeCall(Intent intent) initiates an outgoing call.

Here’s the most typical outgoing call sequence:

  1. OutgoingCallBroadcaster receives a CALL intent and sends the NEW_OUTGOING_CALL broadcast.

  2. The broadcast finally reaches OutgoingCallReceiver, which stashes away a copy of the original CALL intent and launches SipCallOptionHandler.

  3. SipCallOptionHandler decides whether this is a PSTN or SIP call (and in some cases brings up a dialog to let the user choose), and ultimately calls CallController.placeCall() (from the setResultAndFinish() method) with the stashed-away intent from step (2) as the "intent" parameter.

  4. Here in CallController.placeCall() we read the phone number or SIP address out of the intent and actually initiate the call, and simultaneously launch the InCallScreen to display the in-call UI.

  5. We handle various errors by directing the InCallScreen to display error messages or dialogs (via the InCallUiState "pending call status code" flag), and in some cases we also sometimes continue working in the background to resolve the problem (like in the case of an emergency call while in airplane mode). Any time that some onscreen indication to the user needs to change, we update the "status dialog" info in the inCallUiState and (re)launch the InCallScreen to make sure it’s visible.

PhoneUtils

PhoneUtils.placeCall(Context context, Phone phone, String number, Uri contactRef, boolean isEmergencyCall, Uri gatewayUri) dials the number using the phone passed in.

CallManager

CallManager, defined in PhoneApp class.

CallManager class provides an abstract layer for PhoneApp to access and control calls. It implements Phone interface.

CallManager provides call and connection control as well as channel capability.

There are three categories of APIs CallManager provided

  1. Call control and operation, such as dial() and hangup()

  2. Channel capabilities, such as CanConference()

  3. Register notification

public Connection dial(Phone phone, String dialString) in CallManager initiate a new voice connection. This happens asynchronously, so you cannot assume the audio path is connected (or a call index has been assigned) until PhoneStateChanged notification has occurred.

dial() uses PhoneProxy to implement Phone interface to make the call.

Phone Interface

Phone interface specifies the capabilities of the underlying phone system, weather GSM, CDMA, or SIP. We’ll assume GMS from here on.

GSMPhone

GSMPhone extends PhoneBase which in turn implements the Phone interface.

It uses GSMCallTracker to dial via CommandsInterface. This interface is implemented by RIL class.

RIL in turn uses RILRequest to send the requests to rild daemon.

You can use adb logcat -b radio to see the radio-specific log messages.
RIL (Java)

RIL class is the implementation of the CommandsInterface that GSMPhone uses to dial out. This Java implementation writes out the commands to RIL daemon, the native code.

RIL uses RILSender and RILReceiver to send and receive messages from rild - the RIL Daemon. Unlike most of the other Android system services, RIL uses sockets for this inter-process communication, and not the Binder.

RIL Daemon

The RIL consists of two primary components:

Some of the following documentation comes from once available Android Platform Development Kit: Radio Layer Interface documentation. This documentation is no longer readily available from Google (deemed outdated) but is licensed under Apache 2.0 license. We’ve updated the references as originally provided.
RIL Initialization

Android initializes the telephony stack and the Vendor RIL at startup as described in the sequence below:

  1. RIL daemon reads rild.lib path and rild.libargs system properties to determine the Vendor RIL library to use and any initialization arguments to provide to the Vendor RIL.

  2. RIL daemon loads the Vendor RIL library and calls RIL_Init to initialize the RIL and obtain a reference to RIL functions.

  3. RIL daemon calls RIL_register on the Android telephony stack, providing a reference to the Vendor RIL functions

Details of this implementation are available in hardware/ril/rild/rild.c.

RIL Interaction

There are two forms of communication that the RIL handles:

Solicited RIL Commands

The following snippet illustrates the interface for solicited commands:

images/RIL-solicited-commands.png
Example of solicited RIL commands

There are over sixty solicited commands grouped by the following families:

Unsolicited RIL Commands

The following snippet illustrates the interface for unsolicited commands:

images/RIL-unsolicited-commands.png
Example of unsolicited RIL commands

There are over ten unsolicited commands grouped by the following families:

Implementing the RIL

To implement a radio-specific RIL, create a shared library that implements a set of functions required by Android to process radio requests. The required functions are defined in the RIL header (hardware/ril/include/telephony/ril.h).

The Android radio interface is radio-agnostic and the Vendor RIL can use any protocol to communicate with the radio. Android provides a reference Vendor RIL, using the Hayes AT command set, that you can use as a quick start for telephony testing and a guide for commercial vendor RILs. The source code for the reference RIL is found at hardware/ril/reference-ril.

Compile your Vendor RIL as a shared library using the convention libril-<companyname>-<RIL version>.so, for example, libril-acme-124.so, where:

For reference implementation of RIL, see hardware/ril/libril.

Resources

Device Policy on Android

images/DevicePolicyServiceArchitecture.svg
Device Policy Service Architecture

Display on Android

images/SurfaceArchitecture.svg
Surface
images/SurfaceFlingerArchitecture.svg
Surface Flinger

Resources

Camera on Android

See Exposing the Android Camera Stack by Balwinder Kaur and Joe Rickson from Aptina Imaging, Inc.

Resources

NFC on Android

images/NfcServiceArchitecture.svg
NFC Service Architecture
This service is deployed as an application, /system/app/Nfc.apk! The sources can be found at packages/apps/Nfc.

Review Questions

  1. Name at least five application framework services?

  2. What is the most typical communication channel for a client to communicate with its service?

  3. What’s the difference between managers and services?

  4. How do we get the list of all application framework services?

  5. What is the purpose of the servicemanager process?

  6. How do clients/services get access to the servicemanager process?

  7. What role does JNI play in most application framework services?

  8. What is the life-cycle of most application framework services?

  9. What is the relationship between application framework services and daemons?

  10. How do most services communicate with their daemons?

  11. What is the purpose of installd and who uses it and when?

  12. What is the purpose of wpa_supplicant and who uses it?

  13. Unlike the Vibrator (or Power) service, the Alarm service does not have what?

  14. How do services (like Location service) provide call-backs to their clients?

  15. What is AudioFlinger and what is its purpose?

  16. What is the role of Audio Policy Service?

  17. What is Stagefright and what is its purpose?

  18. How do hardware manufactures take advantage of Stagefright?

  19. How does the telephony stack differ from most other services?

  20. What is SurfaceFlinger and what is its purpose?

  21. How does NFC Service differ from most other services?

Customizing Android

The following steps assume building Android JB 4.4.2 for ARMv7 on Ubuntu 12.04 x86_64 or MacOS X 10.6 or 10.7 and that the Android sources are available under a directory pointed to by $AOSP_HOME environment variable. (e.g. export AOSP_HOME=~/aosp/).
The complete code (which we’ll develop in this section) for Marakana Alpha is available at https://github.com/thenewcircle/alpha
The code for Marakana Alpha SDK add-on is available https://github.com/thenewcircle/alpha_sdk_addon

Setting up Custom Device Directory Structure

While we could overlay our device’s custom components over the existing AOSP source tree, that makes it harder to deal with future OS upgrades. Instead, we will create a self-contained directory structure to host our device.

  1. Go to AOSP directory 

    $ cd $AOSP_HOME
    We assume that you set AOSP_HOME to the root of your Android Open Source Project source directory (e.g. ~/aosp/). Unless otherwise stated, the rest of the file paths are assumed to be relative to this directory.

  2. Create our vendor (e.g. marakana) directory 

    $ mkdir device/marakana/

  3. Now create our device (e.g. alpha) sub-directory: 

    $ mkdir device/marakana/alpha

Registering our Device with Android’s Build System

We now want to add our device to the Android’s lunch list

Remember that $ source build/envsetup.sh registers lunch combos that we can later build

  1. Create vendorsetup.sh file for our device (the name of this file is fixed):

    device/marakana/alpha/vendorsetup.sh
    add_lunch_combo full_marakana_alpha-eng
add_lunch_combo full_marakana_alpha-userdebug
add_lunch_combo full_marakana_alpha-user

  2. Re-build the lunch list: 

    $ source build/envsetup.sh
including device/asus/grouper/vendorsetup.sh
including device/asus/tilapia/vendorsetup.sh
including device/generic/armv7-a-neon/vendorsetup.sh
including device/generic/armv7-a/vendorsetup.sh
including device/generic/mips/vendorsetup.sh
including device/generic/x86/vendorsetup.sh
including device/lge/mako/vendorsetup.sh
including device/marakana/alpha/vendorsetup.sh
including device/samsung/maguro/vendorsetup.sh
including device/samsung/manta/vendorsetup.sh
including device/samsung/toro/vendorsetup.sh
including device/samsung/toroplus/vendorsetup.sh
including device/ti/panda/vendorsetup.sh
including sdk/bash_completion/adb.bash

  3. Finally, we can check to see that our device now appear in the lunch menu: 

    $ lunch

You're building on Linux

Lunch menu... pick a combo:
     1. aosp_arm-eng
     2. aosp_x86-eng
     3. aosp_mips-eng
     4. vbox_x86-eng
     5. aosp_grouper-userdebug
     6. aosp_tilapia-userdebug
     7. aosp_deb-userdebug
     8. aosp_flo-userdebug
     9. aosp_mako-userdebug
     10. aosp_hammerhead-userdebug
     11. mini_x86-userdebug
     12. mini_mips-userdebug
     13. mini_armv7a_neon-userdebug
     14. full_marakana_alpha-eng
     15. full_marakana_alpha-userdebug
     16. full_marakana_alpha-user
     17. aosp_manta-userdebug

Which would you like? [aosp_arm-eng]
    • We are not yet ready to select it here, because we have not yet provided the necessary makefiles for full_marakana_alpha. If we do, we’ll get: 
      $ lunch full_marakana_alpha-eng
build/core/product_config.mk:203: *** No matches for product "full_marakana_alpha".  Stop.

** Don't have a product spec for: 'full_marakana_alpha'
** Do you have the right repo manifest?

Adding the Makefile Plumbing for our Device

We now need to add basic support for building our device.

  1. Start by creating a our own AndroidProducts.mk file, which simply defines the actual makefiles to be used when building our device:

    device/marakana/alpha/AndroidProducts.mk
    PRODUCT_MAKEFILES := $(LOCAL_DIR)/full_marakana_alpha.mk
    The only purpose of AndroidProducts.mk file (whose name is fixed) is to set PRODUCT_MAKEFILES to a list of product makefiles to expose to the build system. The only external variable it can use is LOCAL_DIR, whose value will be automatically set to the directory containing this file.

  2. Before we create the full_marakana_alpha.mk makefile, let’s create a common.mk where we’ll configure settings that will be shared between our Alpha device and our Alpha SDK Addon:

    device/marakana/alpha/common.mk
    # Since this file can also be referenced by alpha-sdk_addon
# we cannot assume LOCAL_PATH points to the directory where
# this file is located. Instead, we create another variable
# to capture this directory.
MY_PATH := $(LOCAL_PATH)/../alpha

# Include all makefiles in sub-directories (one level deep)
include $(call all-subdir-makefiles)

  3. Now we are ready to create the main build-file for our device (we call it full_marakana_alpha following the convention of other AOSP device makefiles):

    device/marakana/alpha/full_marakana_alpha.mk
    # Inherit from the emulator product, which defines the base OS
$(call inherit-product, $(SRC_TARGET_DIR)/product/full.mk)

# Discard inherited values and use our own instead
PRODUCT_NAME := full_marakana_alpha
PRODUCT_DEVICE := alpha
PRODUCT_MODEL := Full Marakana Alpha Image for Emulator

# Include the common definitions and packages
include $(LOCAL_PATH)/common.mk
    To create an x86 based platform, inherit from $(SRC_TARGET_DIR)/product/full_x86.mk instead.
    About Product Inheritance

    Our device inherits from the full (a.k.a. emulator) product:

    $(call inherit-product, $(SRC_TARGET_DIR)/product/full.mk)

    The full product (i.e. build/target/product/full.mk) in turn inherits from:

    • build/target/product/full_base_telephony.mk
      • build/target/product/full_base.mk
        • build/target/product/generic_no_telephony.mk
          • build/target/product/core.mk
            • build/target/product/base.mk
      • build/target/product/telephony.mk
    • build/target/board/generic/device.mk

    The combination of these files set up compilation rules that define which modules (a.k.a. PRODUCT_PACKAGES) will get included in the final product (i.e. ROM).

    Real hardware devices also inherit from one of these base products. For example:

    • Galaxy Nexus (device/samsung/maguro/full_maguro.mk) inherits from:
      • build/target/product/full_base_telephony.mk (same as our product)
      • device/samsung/maguro/device.mk (which in turn inherits from device/samsung/tuna/device.mk and many many hardware/vendor-specific products)
    • Nexus 10 (device/samsung/manta/full_manta.mk) inherits from:
      • build/target/product/full_base.mk (same as our product, but no telephony)
      • device/samsung/manta/device.mk (which in turn inherits from many hardware/vendor-specific products)
    Prior to the 4.2 release, Android’s build system also used to define GRANDFATHERED_USER_MODULES in build/core/user_tags.mk, which would automatically get included into every product. These base modules have since been moved to build/target/product/base.mk and are now treated as regular PRODUCT_PACKAGES.

  4. Next, we need to add support for "our" board, so borrow the required BoardConfig.mk and optional system.prop files from the emulator’s "generic" board: 

    $ cp build/target/board/generic/BoardConfig.mk device/marakana/alpha/.
$ cp build/target/board/generic/system.prop device/marakana/alpha/.
    For x86, copy from build/target/board/generic_x86/ instead.
    • system.prop - used to set system-wide properties (here, just RIL settings for the emulator)
    • BoardConfig.mk - defines our device board’s kernel/hardware capabilities:
      device/marakana/alpha/BoardConfig.mk
      …
TARGET_NO_BOOTLOADER := true
TARGET_NO_KERNEL := true
TARGET_ARCH := arm
…
TARGET_ARCH_VARIANT := armv7-a
TARGET_CPU_VARIANT := generic
TARGET_CPU_ABI := armeabi-v7a
TARGET_CPU_ABI2 := armeabi
      About CPU Architecture

      Because we copied build/target/board/generic/BoardConfig.mk we inherited the following:

      TARGET_ARCH := arm
TARGET_ARCH_VARIANT := armv7-a
TARGET_CPU_VARIANT := generic
TARGET_CPU_ABI := armeabi-v7a
TARGET_CPU_ABI2 := armeabi

      Alternatively, we could have built our image for x86, either by copying build/target/board/generic_x86/BoardConfig.mk or by changing our own device/marakana/alpha/BoardConfig.mk to say the following:

      TARGET_CPU_ABI := x86
TARGET_ARCH := x86
TARGET_ARCH_VARIANT := x86

Reporting Device Features

Our system image needs to include one or more XML files in /system/etc/permissions reporting the features supported by the device. Application developers can then:

The frameworks/native/data/etc/ directory contains a set of XML files for all standard Android features. An example of the XML file format is:

<?xml version="1.0" encoding="utf-8"?>
<permissions>
        <!-- This is the standard feature indicating that the device includes WiFi. -->
    <feature name="android.hardware.wifi" />
</permissions>

  1. Add the following to common.mk to indicate that Alpha supports a basic set of handheld features: 

    # These are the hardware-specific features
PRODUCT_COPY_FILES += \
        frameworks/native/data/etc/handheld_core_hardware.xml:system/etc/permissions/handheld_core_hardware.xml

Adding Resource Overlays (Optional)

The Android build system supports a resource overlay feature, which enables us to override specific files in the Android framework/app res/ folders, without having to change anything in the AOSP source directories.

You can overlay only resources, not code files.

As an example, let’s change the default wallpaper for our device (for fun). To do so, we need to overlay frameworks/base/core/res/res/drawable-nodpi/default_wallpaper.jpg file in the final image because that’s what the Launcher2 defaults to.

  1. Make the specific overlay/ directory: 

    $ mkdir -p device/marakana/alpha/overlay/frameworks/base/core/res/res/drawable-nodpi

  2. Push our default_wallpaper.jpg to it:

    device/marakana/alpha/overlay/frameworks/base/core/res/res/drawable-nodpi/default_wallpaper.jpg
    device/marakana/alpha/overlay/frameworks/base/core/res/res/drawable-nodpi/default_wallpaper.jpg:
    This image is also available on GitHub

  3. Finally, we need to tell the build-system to use our overlay/ directory:

    device/marakana/alpha/common.mk:
    …
# Enable overlays
DEVICE_PACKAGE_OVERLAYS := $(MY_PATH)/overlay
Any subsequent changes to the files under DEVICE_PACKAGE_OVERLAYS invalidates all previously built modules that could be affected by our overlay (e.g. applications, framework, etc.). As the result, subsequent builds (i.e. running make) will take much longer.

Generating Our Own Platform Signing Keys (Recommended)

Before we compile our device, we should generate our own platform signing keys [Platform_Keys], otherwise our device will not pass CTS.

  1. Define our subject/issuer info: 

    $ SIGNER="/C=US/ST=California/L=San Francisco/O=Marakana Inc./OU=Android/CN=Android Platform Signer/emailAddress=android@marakana.com"

  2. Remove the existing keys (it does not hurt to back them up first!): 

    $ rm build/target/product/security/*.p*

  3. Generate the platform key: 

    $ echo | development/tools/make_key build/target/product/security/platform "$SIGNER"
creating build/target/product/security/platform.pk8 with no password
Generating RSA private key, 2048 bit long modulus
....................+++
..........................................................+++
e is 3 (0x3)

  4. Generate the shared key: 

    $ echo | development/tools/make_key build/target/product/security/shared "$SIGNER"
creating build/target/product/security/shared.pk8 with no password
Generating RSA private key, 2048 bit long modulus
..................................................................................................+++
............+++
e is 3 (0x3)

  5. Generate the media key: 

    $ echo | development/tools/make_key build/target/product/security/media "$SIGNER"
creating build/target/product/security/media.pk8 with no password
Generating RSA private key, 2048 bit long modulus
...................+++
....................+++
e is 3 (0x3)

  6. Generate the testkey key: 

    $ echo | development/tools/make_key build/target/product/security/testkey "$SIGNER"
creating build/target/product/security/testkey.pk8 with no password
Generating RSA private key, 2048 bit long modulus
....................+++
................................................+++
e is 3 (0x3)

  7. Verify that our keys have been created: 

    $ ls -1 build/target/product/security/*.p*
build/target/product/security/media.pk8
build/target/product/security/media.x509.pem
build/target/product/security/platform.pk8
build/target/product/security/platform.x509.pem
build/target/product/security/shared.pk8
build/target/product/security/shared.x509.pem
build/target/product/security/testkey.pk8
build/target/product/security/testkey.x509.pem

  8. Check that our specific subject/issuer has been used: 

    $ openssl x509 -noout -subject -issuer -in  build/target/product/security/platform.x509.pem
subject= /C=US/ST=California/L=San Francisco/O=Marakana Inc./OU=Android/CN=Android Platform Signer/emailAddress=android@marakana.com
issuer= /C=US/ST=California/L=San Francisco/O=Marakana Inc./OU=Android/CN=Android Platform Signer/emailAddress=android@marakana.com
    The build/target/product/security/.pk8 files are the private keys (.x509.pem are the certificates), and we need to make sure to keep them safe and secure - especially since we did not encrypt them!

Building our Device For The First Time

We are now ready to try out our "custom" device - though there is nothing truly custom yet, except for the PRODUCT_* settings and our own product keys.

  1. For good measure, re-register our device: 

    $ source build/envsetup.sh
including device/marakana/alpha/vendorsetup.sh
…

  2. Now we can lunch of our device: 

    $ lunch full_marakana_alpha-eng

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=4.4.2
TARGET_PRODUCT=full_marakana_alpha
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
TARGET_CPU_VARIANT=generic
HOST_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-3.5.0-34-generic-x86_64-with-Ubuntu-12.04-precise
HOST_BUILD_TYPE=release
BUILD_ID=KOT49H
OUT_DIR=out
============================================

  3. We can now compile our device: 

    $ make -j4
…
Installed file list: out/target/product/alpha/installed-files.txt
Target system fs image: out/target/product/alpha/obj/PACKAGING/systemimage_intermediates/system.img
Install system fs image: out/target/product/alpha/system.img
    Adjust the value of -j based on your system’s available cores, memory, and disk I/O.

  4. Finally, we can run it 

    $ $ANDROID_HOST_OUT/bin/emulator &

  5. And we should see

    screens/MarakanaAlpha-Wallpaper.png
    Custom Wallpaper
    screens/MarakanaAlpha-AboutPhone-v1.png
    Custom Model Number on the About Screen

Adding a Custom Kernel to our Device

Our device would work fine with the provided QEMU-based (i.e. emulator-specific) kernel, but in this section, we will configure a custom kernel ([Android_Building_Linux_Kernel]) so that we can:

About QEMU Kernels

Pre-build QEMU kernels are available in AOSP source tree:

  • Android 4.0 (ICS)
    • prebuilt/android-arm/kernel/kernel-qemu-armv7 for ARMv7
    • prebuilt/android-arm/kernel/kernel-qemu for ARMv5
    • prebuilt/android-x86/kernel/kernel-qemu for x86
  • Android 4.1+
    • prebuilts/qemu-kernel/arm/kernel-qemu-armv7 for ARMv7
    • prebuilts/qemu-kernel/arm/kernel-qemu for ARMv5
    • prebuilts/qemu-kernel/x86/kernel-qemu for x86

The steps outlined here are similar to [Android_Building_Linux_Kernel_for_Emulator]:

  1. Create a directory to host our kernel sources: 

    $ mkdir ~/kernel/
$ cd ~/kernel/

  2. Clone the Goldfish kernel sources: 

    $ git clone https://android.googlesource.com/kernel/goldfish.git
$ cd goldfish/

  3. Check out android-goldfish-3.4 branch of the kernel we wish to build: 

    $ git checkout -t remotes/origin/android-goldfish-3.4

  4. Set the architecture variable to match TARGET_ARCH_VARIANT we set in device/marakana/alpha/BoardConfig.mk: 

    $ export ARCH=arm

    When compiling for x86, set the following instead:

    $ export ARCH=x86

  5. Create the default kernel configuration file (.config): 

    $ make goldfish_armv7_defconfig

    When compiling for ARMv5 or x86, run the following instead:

    $ make goldfish_defconfig

  6. Change the Hardware name from "Goldfish" to "Marakana Alpha Board":

    arch/arm/mach-goldfish/board-goldfish.c:
    …
MACHINE_START(GOLDFISH, "Marakana Alpha Board")
…
    About the Hardware (Board) Name

    As we already know, when the Android kernel loads, it runs init, which configures itself from init.rc and init.hardware_.rc configuration files.

    The hardware name is extracted from /proc/cpuinfo:

    $ adb shell cat /proc/cpuinfo |grep Hardware
Hardware        : Goldfish

    Basically, system/core/init/init.c:main() uses system/core/init/util.c:get_hardware_name(…) to parse /proc/cpuinfo and extract the hardware name. This name is converted to lower-case and all the space characters are trimmed.

    Android’s ueventd process (started by init) also configures itself in a similar way, by loading ueventd.rc and ueventd.hardware.rc configuration files.

    Since in our case we changed the name of our hardware to "Marakana Alpha Board", we will also need to create the corresponding: init.marakanaalphaboard.rc and ueventd.marakanaalphaboard.rc files.

    Alternatively, the hardware name may also be set by the bootloader via kernel boot options. These are exposed to the userspace via /proc/cmdline (e.g. … androidboot.hardware=goldfish …)

  7. Configure the kernel 

    $ make menuconfig
    On Ubuntu 12.04 LTS, you may need to sudo apt-get install lib32ncurses5-dev before you can successfully get to ncurses-based menu config.

    1. Set General setup → Local version to -marakana-alpha-release

    2. Select Enable loadable module support and within it also select Module unloading and within that Forced module unloading

    3. Customize the rest as desired

  8. Set the cross compiler: 

    $ export CROSS_COMPILE=$ANDROID_EABI_TOOLCHAIN/arm-linux-androideabi-

    When compiling for x86, set the following instead:

    $ export REAL_CROSS_COMPILE=$ANDROID_EABI_TOOLCHAIN/i686-linux-android-
$ export CROSS_COMPILE=$AOSP/external/qemu/distrib/kernel-toolchain/android-kernel-toolchain-

  9. Compile the kernel 

    $ make -j4
    About The Build Kernel Script

    AOSP comes with a convenient script to compile the Linux kernel for QEMU-based emulator: $AOSP_HOME/external/qemu/distrib/build-kernel.sh.

    We could use this script instead of setting up our own ARCH and CROSS_COMPILE options and running make.

    For example, to compile for x86, we could do:

    $ $AOSP_HOME/external/qemu/distrib/build-kernel.sh --arch=x86

    The target kernel will go to /tmp/kernel-qemu/ by default.

  10. Copy the compiled kernel to our device’s alpha/ directory: 

    $ cp arch/arm/boot/zImage $AOSP_HOME/device/marakana/alpha/kernel
    For x86, copy arch/x86/boot/bzImage instead.
    In case you were not able to compile the kernel, you could also download a pre-built one (for ARMv7) from https://github.com/thenewcircle/alpha/raw/master/kernel

  11. Now we can go back to the AOSP source 

    $ cd $AOSP_HOME/

  12. Enable our custom kernel in BoardConfig.mk (double-negation, nice!):

    device/marakana/alpha/BoardConfig.mk
    …
TARGET_NO_KERNEL := false
…
    This will cause the build system to rebuild the boot.img file, but because we are still working with the emulator, we’ll have to explicitly reference our kernel on startup.

  13. Now we are ready to create our own init, ueventd, and fstab configuration files by copying the ones from Goldfish: 

    $ cp device/generic/goldfish/init.goldfish.rc device/marakana/alpha/init.marakanaalphaboard.rc
$ cp device/generic/goldfish/ueventd.goldfish.rc device/marakana/alpha/ueventd.marakanaalphaboard.rc
$ cp device/generic/goldfish/fstab.goldfish device/marakana/alpha/fstab.marakanaalphaboard
    Remember, marakanaalphaboard is our device’s new hardware name (lowercased and stripped of spaces).

  14. Next, we need to enable our kernel and copy the these configuration files:

    device/marakana/alpha/common.mk
    …
# Enable our custom kernel
LOCAL_KERNEL := $(MY_PATH)/kernel
PRODUCT_COPY_FILES += $(LOCAL_KERNEL):kernel

# Copy our init and ueventd configuration files to the root
# file system (ramdisk.img -> boot.img)
PRODUCT_COPY_FILES += $(MY_PATH)/init.marakanaalphaboard.rc:root/init.marakanaalphaboard.rc
PRODUCT_COPY_FILES += $(MY_PATH)/ueventd.marakanaalphaboard.rc:root/ueventd.marakanaalphaboard.rc
PRODUCT_COPY_FILES += $(MY_PATH)/fstab.marakanaalphaboard:root/fstab.marakanaalphaboard

  15. Lastly, there is an explicit reference in init.hardware_.rc that points to fstab.hardware that we need to update. In our copied file it points to fstab.goldfish, but we need it to reference our board:

    device/marakana/alpha/init.marakanaalphaboard.rc
    on fs
        mount_all /fstab.marakanaalphaboard
About Storage Mounting

The fstab configuration file used by vold to define volumes and mountpoints for storage has evolved in later Android versions. In Android releases 4.2 and earlier, mounting rules for primary partitions were coded directly into init.rc, and rules for secondary (external) storage were configured via /system/etc/vold.fstab.

system/core/rootdir/init.rc
on fs
# mount mtd partitions
    # Mount /system rw first to give the filesystem a chance to save a checkpoint
    mount yaffs2 mtd@system /system
    mount yaffs2 mtd@system /system ro remount
    mount yaffs2 mtd@userdata /data nosuid nodev
    mount yaffs2 mtd@cache /cache nosuid nodev
system/core/rootdir/etc/vold.fstab
## Example of a standard sdcard mount for the emulator / Dream
# Mounts the first usable partition of the specified device
dev_mount sdcard /mnt/sdcard auto /devices/platform/goldfish_mmc.0 /devices/platform/msm_sdcc.2/mmc_host/mmc1

In Android 4.3 and later, the device-specific configuration rules for mounting all storage were placed into a unified fstab.hardware file. For secondary storage, vold is coded to look specifically for this file, but in order to mount primary storage, the init.hardware.rc file invokes the configuration file directly.

device/generic/goldfish/init.goldfish.rc
on fs
        mount_all /fstab.goldfish
device/generic/goldfish/fstab.goldfish
# Android fstab file.
#<src>                           <mnt_point> <type> <mnt_flags and options>                              <fs_mgr_flags>
# The filesystem that contains the filesystem checker binary (typically /system) cannot
# specify MF_CHECK, and must come before any filesystems that do specify MF_CHECK
/dev/block/mtdblock0             /system     ext4   ro,barrier=1                                         wait
/dev/block/mtdblock1             /data       ext4   noatime,nosuid,nodev,barrier=1,nomblk_io_submit      wait,check
/dev/block/mtdblock2             /cache      ext4   noatime,nosuid,nodev,nomblk_io_submit,errors=panic   wait,check
/devices/platform/goldfish_mmc.0 auto        vfat   defaults                                             voldmanaged=sdcard:auto

  1. Re-build our ROM, which, in this case just creates a new out/target/product/alpha/boot.img (that we won’t actually use): 

    $ make -j10
…

  2. Restart the emulator (with our new kernel) 

    $ $ANDROID_HOST_OUT/bin/emulator -kernel out/target/product/alpha/kernel &
    We have to run our emulator with -kernel, because otherwise it would ignore the kernel in our newly built boot.img and instead use the default one (prebuilt/android-arm/kernel/kernel-qemu-armv7). This applies just to the emulator.

  3. Test 

    $ adb shell cat /proc/version
Linux version 3.4.0 (student@newcircle) (gcc version 4.7 (GCC) ) #4 PREEMPT Sun Feb 16 22:46:38 PST 2014
    The path to adb was added to our PATH when we $ source build/envsetup.sh (on a Linux host, it comes from out/host/linux-x86/bin/adb)

  4. We could also take a look at the updated About screen:

    screens/MarakanaAlpha-AboutPhone-v2.png

Adding a Custom Native Library and Executable to our Device

Using our Native Library via a Custom Daemon

Exposing our Native Library via Java (i.e. JNI)

Consuming our a Custom Java/JNI→Native Library via a Custom App (Optional)

Exposing our Custom Library via a Custom IPC/Binder Service

Rather than have 3rd party applications direct access to "our" driver (via JNI/HAL), we may be better off providing access to the driver’s functionality via a Binder (AIDL-described) service.

Building a Custom App Using a Custom Service Manager

Creating a Custom SDK Add-on (Optional)

Building SDK Components

In order to build an SDK Add-on, we need to also build the full Android SDK. By default, repo init does not pull all the packages necessary to build the SDK or related components. Any package that is part of the notdefault group is not part of the sync, and this includes the packages for the Android SDK.

platform/manifest/default.xml
<manifest>
  …
  <project path="tools/adt/eclipse" name="platform/tools/adt/eclipse" groups="notdefault,tools" />
  <project path="tools/adt/idea" name="platform/tools/adt/idea" groups="notdefault,tools" />
  <project path="tools/base" name="platform/tools/base" groups="notdefault,tools" />
  …
</manifest>

In order to pull all the packages necessary to build the SDK (and Add-ons), append -g all to the end of the repo init command:

$ repo init -u https://android.googlesource.com/platform/manifest -b android-4.4.2_r1 -g all
$ repo sync

Distributing our Custom SDK Add-on (Optional)

  1. First, we need to create a repository.xml file to describe our add-on and publish it to our server:

    https://marakana.com/external/android/sdk-addon/repository.xml
    <?xml version="1.0" encoding="UTF-8"?>

<sdk:sdk-addon
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:sdk="http://schemas.android.com/sdk/android/addon/1">

    <sdk:add-on>
        <sdk:name>Alpha Add-On</sdk:name>
        <sdk:api-level>15</sdk:api-level>
        <sdk:vendor>Marakana</sdk:vendor>
        <sdk:revision>1</sdk:revision>
        <sdk:description>Android + Marakana Alpha Add-on, API 15, revision 1 </sdk:description>
        <sdk:desc-url>http://marakana.com/external/android/sdk-addon/</sdk:desc-url>
        <sdk:uses-license ref="marakana-android-addon-license" />
        <sdk:archives>
            <sdk:archive os="any">
                <sdk:size>96382546</sdk:size>
                <sdk:checksum type="sha1">5cec8cc3f3064441cb96a50e2b8aa528681ffc78</sdk:checksum>
                <sdk:url>marakana_alpha_sdk_addon_api-15_r1.zip</sdk:url>
            </sdk:archive>
        </sdk:archives>
        <sdk:libs>
        </sdk:libs>
    </sdk:add-on>

    <sdk:license type="text" id="marakana-android-addon-license">
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this SDK addon except in compliance with the License.
You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
    </sdk:license>
</sdk:sdk-addon>
    We have to make sure that we adjust the info as necessary to reflect our particular site/addon.

  2. Next, we need to upload our addon to our site, relative to <sdk:sdk-addon> → <sdk:add-on> → <sdk:desc-url> URL (e.g. https://marakana.com/external/android/sdk-addon/marakana_alpha_sdk_addon_api-15_r1.zip)

  3. We are now ready to test it out in our Android SDK Manager

    1. Go to Tools → Manage Add-on Sites… in the menu bar

    2. Click on New…

    3. Enter the base URL in the URL field (e.g. https://marakana.com/external/android/sdk-addon/) and click on OK

    4. Click on Close to dismiss the Add-on Sites window

    5. Under Packages we should now see Alpha Add-On by Marakana, with a status Not installed

    6. Click on the checkbox next to our package

    7. Click on Install 1 package… button

    8. Accept the license terms and click on Install

  4. Test that it works by creating an AVD based on our add-on and/or using it in Eclipse

Review Questions

  1. What is a home directory for a new custom board within AOSP source-tree?

  2. What is the purpose of vendorsetup.sh?

  3. What does $(call inherit-product, $(SRC_TARGET_DIR)/product/generic.mk) do?

  4. What is the significance of PRODUCT_PACKAGES?

  5. What is the purpose of BoardConfig.mk?

  6. What is the purpose of development/tools/make_key and why do we want to use it?

  7. What is the significance of /proc/cpuinfo?

  8. When/why do we use include $(call all-subdir-makefiles)?

  9. What is the purpose of LOCAL_MODULE?

  10. What is the purpose of LOCAL_MODULE_TAGS?

  11. How do we compile individual modules efficiently?

  12. Name at least three include $(BUILD_XXX) targets.

  13. What do we need to do in order to launch a custom daemon on startup?

  14. How do we expose custom Java system libraries to applications?

  15. How do applications consume custom Java system libraries?

  16. When do we use include $(BUILD_PREBUILT)?

  17. When/where do we use $(JNI_H_INCLUDE)?

  18. What is the purpose of {@hide}?

  19. When do we use include $(BUILD_DROIDDOC)?

  20. What do we need to do in order to register a custom service to servicemanager?

  21. When/why would we use LOCAL_JAVA_LIBRARIES += framework?

  22. When/why would we use LOCAL_CERTIFICATE := platform?

  23. What does include $(BUILD_PACKAGE) build?

  24. What is SDK add-on and when/why would we want to build/use one?

  25. How do we distribute custom SDK add-ons?

Android Tools and Debugging

Objectives of Tools and Debugging Module

Android is a complex system, composed of many moving parts and layers. In this module, we will explore Android’s many tools and configuration options that will enable us to:

Kernel Debugging

Logging

Stack Traces

Debugging Native Code with GDB

  1. Map a port on our host machine to a port on the emulator or device that GDB will listen on: 

    $ adb forward tcp:5039 tcp:5039
    About Port Mapping

    Here, we use port 5039 because it is what gdbclient helper function uses (provided by build/envsetup.sh), although the "official" port for GDB is 2159, according IANA.

    When this is done, we’ll have 127.0.0.1:5039 on our host machine → 0.0.0.0:5039 on the emulator/device.

    Emulator also permits port-mapping via its management console (assuming it’s accessible via 5554):

    $ echo "redir add tcp:5039:5039" | nc localhost 5554

  2. Debug a new program or an existing process with the help of the GDB server:

    • Get GDB server to start a new process by specifying its path and arguments. For example: 
      $ adb shell gdbserver :5039 /system/bin/mrknlog &
      Here, we are starting a (custom) program (mrknlog) that prints the used/total size of /dev/log/main buffer and flushes it before exiting.
    • Get GDB server to attach to an existing process via its PID. For example: 
      $ pid mrknlogd
47
$ adb shell gdbserver :5039 --attach 47 &
Attached; pid = 47
Listening on port 5039
      Here, we are connecting to a (custom) daemon process (mrknlogd), which does exactly what mrknlog does, except repeatedly, while sleeping between executions. We assume mrknlogd is already running (e.g. started by init).

      The shell function pid mrknlogd (provided by build/envsetup.sh) is equivalent to:

      $ adb shell ps | grep mrknlogd  | awk '{print $2}'
47

  3. Get the location of unstripped binaries: 

    $ export SYMBOLS=$ANDROID_PRODUCT_OUT/symbols

    Here, ANDROID_PRODUCT_OUT is set by the lunch command.

    Alternatively, we could also do:

    $ export SYMBOLS=`get_abs_build_var TARGET_OUT_UNSTRIPPED`
    About Debugger Symbols

    To save on (flash-memory) space, Android by default strips all of the symbols from its executables and shared libraries. While symbols (i.e. the debug info) are not needed at run-time, GDB uses the symbol table to resolve line numbers as well as function and variable names. We’d be lost trying to debug without these.

    So, what do we do?

    We could provide GDB with location of unstripped binaries (the direction we’ll take).

    Or, we could re-compile our binaries and instruct the build system not to strip the symbol information (as well as disable compiler optimizations), by placing the following in our Android.mk file(s):

    LOCAL_CFLAGS += -g -O0

    Replace LOCAL_CFLAGS with LOCAL_CPPFLAGS in case of C++.

  4. Connect with the debugger client (gdb) to the debugger server (gdbserver): 

    $ANDROID_TOOLCHAIN/arm-linux-androideabi-gdb \
  -ex "set solib-absolute-prefix $SYMBOLS" \
  -ex "set solib-search-path $SYMBOLS/system/lib:$SYMBOLS/system/lib/hw:$SYMBOLS/system/lib/ssl/engines" \
  -ex "target remote :5039" \
  --quiet \
  $SYMBOLS/system/bin/mrknlog
Reading symbols from /home/student/aosp/out/target/product/alpha/symbols/system/bin/mrknlog...done.
Remote debugging using :5039
Remote debugging from host 127.0.0.1
__dl__start () at bionic/linker/arch/arm/begin.S:35
35              mov     r0, sp
(gdb)
    For x86 targets, run $ANDROID_EABI_TOOLCHAIN/i686-linux-android-gdb instead. Replace mrknlog with mrknlogd when attaching to our mrknlogd daemon.
    Android ships with a gdbclient bash function (courtesy of build/envsetup.sh) that tries to automate all of these steps, though its usage is somewhat confusing.
    You may find GDB cheatsheet handy.
    About the GDB Client

    Here we are running gdb client from the Android-provided arch-specific toolchain.

    Using the -ex command-line options, we are asking gdb to:

    1. Initialize the system root to the root of our un-stripped binaries

    2. Initialize the search path for shared objects to the locations of our unstripped shared libraries

    3. Connect to the GDB server on the previously mapped port (127.0.0.1:5039)

    Finally, we are also providing gdb with the unstripped version of the executable we are debugging.

  5. Start debugging:

    • Debugging a new program, starting from main(…): 
      (gdb) break main
Breakpoint 1 at 0x2a0004e0: file device/marakana/alpha/bin/mrknlog/mrknlog.c, line 7.
(gdb) continue
Continuing.

Breakpoint 1, main (argc=1, argv=0xbee96cb4) at device/marakana/alpha/bin/mrknlog/mrknlog.c:7
7       int main (int argc, char* argv[]) {
(gdb) list
2       #include <string.h>
3       #include <errno.h>
4
5       #include <mrknlog.h>
6
7       int main (int argc, char* argv[]) {
8           int usedSize = mrkn_get_used_log_size();
9           int totalSize = mrkn_get_total_log_size();
10          if (totalSize >= 0 && usedSize >= 0) {
11              if (mrkn_flush_log() == 0) {
(gdb) b 11
Breakpoint 2 at 0x2a0004fe: file device/marakana/alpha/bin/mrknlog/mrknlog.c, line 11.
(gdb) c
Continuing.

Breakpoint 2, main (argc=<optimized out>, argv=<optimized out>)
    at device/marakana/alpha/bin/mrknlog/mrknlog.c:11
11              if (mrkn_flush_log() == 0) {
(gdb) print usedSize
$1 = 223
(gdb) bt
#0  main (argc=<optimized out>, argv=<optimized out>) at device/marakana/alpha/bin/mrknlog/mrknlog.c:11
…
    • Debugging an existing process (mrknlogd): 
      …
Reading symbols from /home/student/aosp/out/target/product/alpha/symbols/system/bin/mrknlogd...done.
Remote debugging using :5039
Remote debugging from host 127.0.0.1
warning: .dynamic section for "/home/student/aosp/out/target/product/alpha/symbols/system/bin/linker" is not at the expected address (wrong library or version mismatch?)
nanosleep () at bionic/libc/arch-arm/syscalls/nanosleep.S:10
10          ldmfd   sp!, {r4, r7}
(gdb) bt
#0  nanosleep () at bionic/libc/arch-arm/syscalls/nanosleep.S:10
#1  0x40043294 in sleep (seconds=<optimized out>) at bionic/libc/unistd/sleep.c:45
#2  0x2a0005e0 in main (argc=<optimized out>, argv=<optimized out>)
    at device/marakana/alpha/bin/mrknlogd/mrknlogd.c:28
(gdb) frame 2
#2  0x2a0005e0 in main (argc=<optimized out>, argv=<optimized out>)
    at device/marakana/alpha/bin/mrknlogd/mrknlogd.c:28
28            sleep(frequency);
(gdb) print frequency
$1 = 60
(gdb) set frequency=10
(gdb) print frequency
$2 = 60
(gdb) break 27
Breakpoint 1 at 0x2a0005da: file device/marakana/alpha/bin/mrknlogd/mrknlogd.c, line 27.
(gdb) c
Continuing.

Breakpoint 1, main (argc=<optimized out>, argv=<optimized out>)
    at device/marakana/alpha/bin/mrknlogd/mrknlogd.c:28
28            sleep(frequency);
(gdb) print usedSize
$6 = 415
(gdb) detach
Detaching from process 47
Ending remote debugging.
(gdb) quit
$
    • Debugging a shared library within an existing Java (Dalvik) process (com.marakana.android.logservice): 
      $ adb shell ps | grep system.*com.marakana.android.logservice | awk '{print $2}'
300
$ adb shell gdbserver :5039 --attach 300 &
Attached; pid = 300
Listening on port 5039
$ $ANDROID_TOOLCHAIN/arm-linux-androideabi-gdb \
  -ex "set solib-absolute-prefix $SYMBOLS" \
  -ex "set solib-search-path $SYMBOLS/system/lib:$SYMBOLS/system/lib/hw:$SYMBOLS/system/lib/ssl/engines" \
  -ex "set breakpoint pending on" \
  -ex "target remote :5039" \
  --quiet \
  $SYMBOLS/system/bin/app_process
Reading symbols from /home/student/aosp/out/target/product/alpha/symbols/system/bin/app_process...done.
Remote debugging using :5039
Remote debugging from host 127.0.0.1
warning: .dynamic section for "/home/student/aosp/out/target/product/alpha/symbols/system/bin/linker" is not at the expected address (wrong library or version mismatch?)
epoll_wait () at bionic/libc/arch-arm/syscalls/epoll_wait.S:10
10          ldmfd   sp!, {r4, r7}
(gdb) b device/marakana/alpha/lib/libmrknlog/libmrknlog.c:22
Breakpoint 1 at 0x49f96504: file device/marakana/alpha/lib/libmrknlog/libmrknlog.c, line 22.
(gdb) c
Continuing.
[New Thread 1171]
[Switching to Thread 1171]

Breakpoint 1, ioctl_log (mode=<optimized out>, request=44546)
    at device/marakana/alpha/lib/libmrknlog/libmrknlog.c:24
24      }
(gdb) print request
$1 = 44546
(gdb) detach
(gdb) quit
      About Debugging Dalvik Processes

      Debugging native-code in any Dalvik-based (or Zygote-forked) process requires that we start from system/bin/app_process, because that’s what zygote first runs. In order to reference code in shared libraries (yet to be loaded), we need enable pending break points.

      We can either do this via -ex "set breakpoint pending on" command-line switch, or in GDB when it starts:

      (gdb) set breakpoint pending on

      Debugging Java code will be covered later.

    • Debugging a shared library within System Server 
      $ pid system_server
307
$ adb shell gdbserver :5039 --attach 307 &
Attached; pid = 307
Listening on port 5039
$ $ANDROID_TOOLCHAIN/arm-linux-androideabi-gdb \
  -ex "set solib-absolute-prefix $SYMBOLS" \
  -ex "set solib-search-path $SYMBOLS/system/lib:$SYMBOLS/system/lib/hw:$SYMBOLS/system/lib/ssl/engines:$SYMBOLS/system/vendor/lib/hw:$ANDROID_PRODUCT_OUT/obj/lib" \
  -ex "set breakpoint pending on" \
  -ex "target remote :5039" \
  --quiet \
  $SYMBOLS/system/bin/app_process
Reading symbols from /Volumes/Android/aosp-4.1/out/target/product/maguro/symbols/system/bin/app_process...done.
Remote debugging using :5039
Remote debugging from host 127.0.0.1
warning: .dynamic section for "/Volumes/Android/aosp-4.1/out/target/product/maguro/symbols/system/bin/linker" is not at the expected address (wrong library or version mismatch?)
__ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:9
9           swi     #0
(gdb) break hardware/libhardware_legacy/vibrator/vibrator.c:61
Breakpoint 1 at 0x400ecc3c: file hardware/libhardware_legacy/vibrator/vibrator.c, line 61.
(gdb) c
Continuing.

Breakpoint 1, sendit (timeout_ms=20) at hardware/libhardware_legacy/vibrator/vibrator.c:61
61          close(fd);
(gdb) bt
#0  sendit (timeout_ms=20) at hardware/libhardware_legacy/vibrator/vibrator.c:61
#1  0x400ecc8c in vibrator_on (timeout_ms=20) at hardware/libhardware_legacy/vibrator/vibrator.c:69
#2  0x56f90d3c in android::vibratorOn (env=0x5c514d30, clazz=0x40200009, timeout_ms=20) at frameworks/base/services/jni/com_android_server_VibratorService.cpp:40
#3  0x4073de34 in dvmPlatformInvoke () at dalvik/vm/arch/arm/CallEABI.S:258
#4  0x4076d086 in dvmCallJNIMethod (args=0x5bbe1794, pResult=0x58000400, method=0x572856d0, self=0x580003f0) at dalvik/vm/Jni.cpp:1155
…
#23 0x407743d6 in interpThreadStart (arg=0x580003f0) at dalvik/vm/Thread.cpp:1538
#24 0x40029bb4 in __thread_entry (func=0x40774335 <interpThreadStart(void*)>, arg=0x580003f0, tls=<optimized out>) at bionic/libc/bionic/pthread.c:217
#25 0x4002930c in pthread_create (thread_out=0x5c3f1e28, attr=0x5e2bed70, start_routine=0x40774335 <interpThreadStart(void*)>, arg=0x580003f0) at bionic/libc/bionic/pthread.c:356
#26 0x00000000 in ?? ()
(gdb) frame 2
#2  0x56f90d3c in android::vibratorOn (env=0x5c514d30, clazz=0x40200009, timeout_ms=20) at frameworks/base/services/jni/com_android_server_VibratorService.cpp:40
40          vibrator_on(timeout_ms);
(gdb) list
35      }
36
37      static void vibratorOn(JNIEnv *env, jobject clazz, jlong timeout_ms)
38      {
39          // ALOGI("vibratorOn\n");
40          vibrator_on(timeout_ms);
41      }
42
43      static void vibratorOff(JNIEnv *env, jobject clazz)
44      {
(gdb) print timeout_ms
$1 = 20
(gdb) detach
(gdb) quit

      In this case, we disabled:

      Otherwise, we would have had harder time walking through the code we were interested in.
      About Debugging with Proprietary Binaries

      When debugging on real hardware devices, we should also include proprietary binaries (mostly HAL libraries) into our solib-search-path.

      These binaries are not under $ANDROID_PRODUCT_OUT/symbols/*, because they come pre-built.

      All we need to do is append $SYMBOLS/system/vendor/lib/hw:$ANDROID_PRODUCT_OUT/obj/lib to solib-search-path.

      About Debugging Multithreaded Processes

      Most processes on Android are multi-threaded, but GDB can only step through one thread at a time. In order to prevent (suspend) threads from running while stepping through code, we can:

      (gdb) set scheduler-locking on

      We just have to remember to either continue or disable the lock, in order to allow for normal execution. Otherwise, we could cause a dead-lock.

      (gdb) set scheduler-locking off

      For more info, see Debugging Programs with Multiple Threads

Debugging Android Applications

Debugging Existing Java Applications/Processes

Debugging ActivityManagerService Example

$ adb forward tcp:8700 jdwp:$(pid system_server)
jdb -attach 127.0.0.1:8700
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...
> stop in com.android.server.am.ActivityManagerService.updateOomAdjLocked()
Set breakpoint com.android.server.am.ActivityManagerService.updateOomAdjLocked()
> stop in com.android.server.am.ActivityManagerService.updateOomAdjLocked(
Breakpoint hit: "thread=12 android.server.ServerThread", com.android.server.am.ActivityManagerService.updateOomAdjLocked(), line=14,705 bci=0

> stop at com.android.server.am.ActivityManagerService:14731
Set breakpoint com.android.server.am.ActivityManagerService:14731
12 android.server.ServerThread[1] cont
12 android.server.ServerThread[1] where
  [1] com.android.server.am.ActivityManagerService.updateOomAdjLocked (ActivityManagerService.java:14,731)
  [2] com.android.server.am.BroadcastQueue.deliverToRegisteredReceiverLocked (BroadcastQueue.java:418)
  [3] com.android.server.am.BroadcastQueue.processNextBroadcast (BroadcastQueue.java:634)
  [4] com.android.server.am.ActivityManagerService.finishReceiver (ActivityManagerService.java:13,281)
  [5] android.content.BroadcastReceiver$PendingResult.sendFinished (BroadcastReceiver.java:417)
  [6] android.content.BroadcastReceiver$PendingResult.finish (BroadcastReceiver.java:393)
  [7] android.app.LoadedApk$ReceiverDispatcher$Args.run (LoadedApk.java:772)
  [8] android.os.Handler.handleCallback (Handler.java:615)
  [9] android.os.Handler.dispatchMessage (Handler.java:92)
  [10] android.os.Looper.loop (Looper.java:137)
  [11] com.android.server.ServerThread.run (SystemServer.java:875)
12 android.server.ServerThread[1] print numSlots
numSlots = 7
12 android.server.ServerThread[1] print factor
factor = 2
12 android.server.ServerThread[1] print i
i = 21
12 android.server.ServerThread[1] quit
Recall that pid is a function exposed by build/envsetup.sh.

Service Binaries

Digging into System Services

Legal Disclaimer

INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. NEWCIRCLE ASSUMES NO LIABILITY WHATSOEVER AND NEWCIRCLE DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

Other names and brands may be claimed as the property of others.

Copyright © 2014 NewCircle, Inc. This deck was originally written and copyrighted by Marakana, Inc and licensed to NewCircle by Twitter, Inc.

Marakana Authors

NewCircle Authors for Kit Kat Updates

/

#